Event-driven spiking convolutional neural networks

Abstract

The present invention relates to an event-driven spiking convolutional neural network comprising a plurality of layers, each layer comprising: - a kernel module configured to store and process kernel values ​​of at least one convolution kernel in an event-driven manner; - a neuron module configured to store and process neuron states of neurons of the neural network in an event-driven manner and to output spike events generated from the updated neurons; - a memory mapper configured to determine a neuron to which an incoming spike event from a source layer projects by convolution with at least one convolution kernel, wherein the neuron state of the determined neuron is updated with applicable kernel values ​​of the at least one convolution kernel, and the memory mapper (130) is configured to process the incoming spike event in an event-driven manner.

Description

[Technical field]

[0001] The present invention relates to an event-driven spiking convolutional neural network, a computer program for controlling said event-driven spiking convolutional neural network, and a memory mapper for said event-driven convolutional neural network. [Background technology]

[0002] Convolutional neural networks (CNNs) and corresponding chip architectures for efficiently performing CNN tasks are known in the art.

[0003] A particular class of CNNs includes spiking convolutional networks (sCNNs), which mimic the function of biological neurons in terms of discrete signal behavior in the form of electrical spikes generated when a particular membrane potential of the neuron is reached, in contrast to traditional CNNs, which provide a continuous output for any input, making them less energy efficient than sCNNs.

[0004] Secondly, although sCNN is mathematically well described, no implementations in dedicated hardware are known: sCNN implemented on general-purpose processors are either relatively slow or have particularly high memory requirements, thus losing the advantages of its specific mode of data processing.

[0005] However, currently there are no dedicated electronic circuits specifically designed to run sCNN.

[0006] A general (not necessarily convolutional) neural network is taught by IBM et al.'s "true north" chip architecture (US Patent 2014 / 0032465 A1) which is based on a so-called crossbar architecture that provides the ability to interconnect each neuron of the neural network with each other in almost any manner. However, this can impose very large memory requirements, since each connection between neurons has an associated weight. Thus, the memory requirements of a weight matrix containing all the weights, since each neuron is interconnected with every other neuron, is roughly proportional to the square of the number of neurons and quickly becomes infeasible.

[0007] Furthermore, apart from the True North architecture, known processors for CNNs operate in a frame-based manner, which is opposite to the mode of signal processing that is advantageous when dealing with sCNNs (frame-based vs. event-based).

[0008] Currently, there is no electronic circuit architecture specifically designed to run sCNN efficiently and quickly. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] US Patent No. 2014 / 0032465 A1 Summary of the Invention [Problem to be solved by the invention]

[0010] It is an object of the present invention to provide a system for executing spiking convolutional networks in an energy and memory efficient manner. This object is achieved by an apparatus having the features of claim 1. [Means for solving the problem]

[0011] Advantageous embodiments are set forth in the dependent claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic layout of one layer of an sCNN according to the present invention. [Diagram 2] Schematic layout of an sCNN containing multiple layers. [Diagram 3] FIG. 2 is a schematic data flow diagram of a memory mapper. [Figure 4] FIG. 2 is a schematic diagram showing how neuron coordinates and applicable kernel values ​​are determined. [Diagram 5] FIG. 1 is a schematic data flow diagram showing an update process of a neuron. [Figure 6] FIG. 1 is a diagram of a parallelized neuron module architecture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the term "spiking convolutional neural network" and similar terms refer to at least one electronic circuit constructed and arranged to operate a spiking convolutional neural network, unless otherwise stated or implied.

[0014] According to claim 1, an event-driven spiking convolutional neural network (sCNN) comprises a number of layers, each layer comprising: - a kernel module configured to store and process kernel values ​​of at least one convolution kernel in an event-driven manner; a neuron module adapted to store and process in an event-driven manner neuronal states of the neurons of said network, in particular to update and to output spike events generated from the specifically processed, e.g. updated, neurons; - a memory mapper configured to determine a neuron to which an incoming spike event from a source layer projects by convolution with at least one convolution kernel, where a neuron state of the determined neuron is updated with applicable kernel values ​​of the at least one convolution kernel, the memory mapper being configured to process the incoming spike event in an event-driven manner.

[0015] An event-driven sCNN is an electronic circuit and / or at least one computer chip that includes components such as a memory mapper, a neuron module, and a kernel module, each of which has hardwired communication with the electronic circuit or at least one computer chip.

[0016] That is, while some components of an sCNN may be programmable, the components are not merely reflected or implemented in a computer program or software that is then executed, for example, on a general purpose chip (i.e., the components of an sCNN are not virtual and do not function on a general purpose computer chip), but are the physical embodiment of the electronic circuitry of an sCNN.

[0017] The term "layer" refers in particular to at least one electronic circuit that represents a layer of an sCNN, in particular a layer in the general terminology of a CNN.

[0018] The event-driven sCNN according to the present invention in particular comprises a limited and predefined number of layers.

[0019] Unlike traditional CNNs, the components of an event-driven sCNN are configured to process information in an event-driven manner.

[0020] The events that drive sCNN are given in particular by spike events received by one of the layers of sCNN.

[0021] A spike event is digital data that contains, among other things, structured information about the event.

[0022] Thus, in contrast to other CNNs, event-driven sCNNs are particularly well suited to complying with data streams such as those provided by, for example, event-generating dynamic vision cameras and other event-generating devices, since they operate in a non-frame- or clock-based manner.

[0023] The kernel module stores the kernel value of at least a single convolution kernel (sometimes referred to herein simply as a "kernel") that can be specifically applied to every incoming spike event.

[0024] This allows a kernel module to maintain a similarly small associated kernel memory to store the kernel values ​​of at least one convolution kernel.

[0025] In particular, in contrast to general neural networks, the presence of at least one convolution kernel reduces the memory requirements of sCNN since its neuronal connections feature a smaller set of weights.

[0026] In event-driven sCNN, it is common for a kernel module to store multiple convolution kernels.

[0027] The neuron module essentially contains all the neurons of a layer, which are specifically associated with the output feature map of the layer.

[0028] Each neuron is specifically included in an output feature map, and the position of a neuron in the output feature map is specifically given by neuron coordinates.

[0029] The output feature map is specifically a two-dimensional array of values, said values ​​corresponding to the neuron states of the neurons.

[0030] The term "neuron" refers specifically to an entity of an sCNN characterized in that the sCNN includes adjustable neuronal states, which refer to functions that characterize when and how the neuron outputs spike events.

[0031] Moreover, the neuron module operates in an event-driven manner.

[0032] In particular, with each incoming spike event, at least some neurons of the neuronal module are updated (i.e., processed) with certain kernel values ​​and, depending on the state of the neuron (sometimes referred to in the art as the membrane potential), output the spike event itself.

[0033] The update process specifically involves adding or subtracting the neuron states stored at the respective neuron addresses with calculated values, including the applicable kernel values ​​and, in particular, bias values.

[0034] It should be noted that the neuron module is configured to store and process neuron states of the neurons of the network, and depending on the updated neuron states, the neuron module may output one or more spike events generated from the updated neurons.

[0035] In general, not every updated neuron will necessarily output a spike event.

[0036] According to another embodiment of the invention, the kernel module comprises a kernel interface connected to an associated kernel memory, said associated kernel memory configured to store kernel values ​​of at least one convolution kernel, in particular a plurality of convolution kernels, at kernel addresses, said kernel module being electronically connected to the neuron module.

[0037] The associated kernel memory comprises or is specifically memory physically provided by each layer, e.g., located on each layer, which has the advantage that each layer can be manufactured identically and operate as an autonomous unit.

[0038] Alternatively, or in addition, the associated kernel memory may comprise or be memory allocated to and connected to said layer, but need not be specifically provided by said layer. The associated kernel memory may comprise a global, specifically external, kernel memory connected to a kernel module of said layer.

[0039] In one embodiment, all layers of an sCNN are connected to a global kernel memory, which comprises an associated kernel memory for each layer.

[0040] A kernel address specifically refers to a kernel memory address where a single kernel value is stored.

[0041] According to another embodiment of the invention, the neuron module comprises a neuron memory, said neuron memory adapted to store the state of a neuron in a neuron address.

[0042] According to another embodiment, the neuronal memory comprises a plurality of neuronal memory units accessible in a parallel manner, such neuronal memory units are also referred to herein as associated neuronal memories.

[0043] The neuron memory, in particular each associated neuron memory, may comprise or be, for example, a memory storage physically provided in each layer, located therein, or a memory assigned to and connected to said layer, but not necessarily provided by said layer in particular. Additionally or alternatively, the associated neuron memory may be provided in a global, in particular external, neuron memory connected to the neuron module of the layer.

[0044] In one embodiment, all layers of an sCNN are connected to a global neuron memory, which comprises an associated neuron memory for each layer.

[0045] It is also possible that the global kernel memory and the global neuron memory are provided by a single memory component.

[0046] A neuron address specifically refers to an address in a neuron memory where the neuron state of a certain neuron is stored.

[0047] According to another embodiment of the invention, a memory mapper is electronically connected to a kernel module, in particular to a kernel interface, said memory mapper being configured to: a) receiving an incoming, particularly single, spike event from a source layer of the sCNN via an electronic connection, where the incoming spike event includes information about the coordinates of a particular single neuron in the source layer; and in response to the received spike event: b) determining neuron coordinates and corresponding neuron addresses in the associated neuron memory, in particular of the neurons onto which a convolution of the received incoming spike event with at least one convolution kernel projects; c) determining kernel coordinates using corresponding kernel addresses of kernel values ​​applicable to the neuron state of the determined neuron address, where the neuron state of the determined neuron address is updated with the applicable kernel values, in particular from the at least one convolution kernel or from the plurality of convolution kernels; d) providing the determined kernel addresses for the applicable kernel values, and in particular the determined neuron addresses, to the kernel module, in particular to the kernel interface; The present invention is configured and arranged as follows.

[0048] A memory mapper is in particular an electronic circuit or a system of electronic circuits.

[0049] In particular, the memory mapper calculates where an incoming spike event projects to, this calculation being based in particular on a fully hard-wired formula that allows the memory mapper to determine the neuron address to update and the kernel address with the appropriate kernel value.

[0050] To determine the neuron addresses and kernel addresses, the memory mapper may be provided with programmable register values ​​that specify the formulas for said determination.

[0051] The memory mapper is an asynchronous serial interface circuit with a parallel interface of a given bandwidth, in particular so that it is event-driven, in particular so that it can process at least one spike event at a time.

[0052] Memory mappers are especially SRAM modules, flash memories, etc.

[0053] The source layer is in particular provided by an sCNN. The source layer can also be the current layer, for example if spike events generated by a neuron module are rerouted to the memory mapper of the layer.

[0054] The term "projects to" and similar terms specifically refer to essentially the inverse of the receptive field of a CNN.

[0055] Because sCNN is event-driven, the convolution operations can be performed particularly efficiently on single spike events.

[0056] According to another embodiment of the present invention, the kernel module is configured and arranged to receive the determined kernel address from a specifically associated kernel memory and to output the applicable kernel value stored at the determined kernel address to the neuron module together with the specifically determined neuron address.

[0057] According to another embodiment of the invention, the neuron module comprises: a) receiving a determined neuron address and an applicable kernel value; b) for each received neuron address, determining an updated neuron state for that neuron based on the applicable kernel value; c) outputting an outgoing spike event when, in particular after the neuron state has been updated, the updated neuron reaches, i.e. in particular exceeds or falls below, at least one predefined threshold value; The system is constructed and arranged as follows.

[0058] That is, when the neuron state reaches a certain threshold, a spike event occurs.

[0059] The neuron states are specifically represented by numbers.

[0060] The term "reaching" specifically refers to a neuronal state going below or above a threshold.

[0061] According to another embodiment of the present invention, the memory mapper is constructed and arranged to determine, for each received incoming spike event, at least one output feature map of neurons assigned to the output feature map, the number of said output feature maps being equal to the number of convolution kernels of the current layer.

[0062] A feature map can be represented as a two-dimensional array of neurons addressable by their neuronal coordinates. Each neuron has a neuron state.

[0063] An incoming spike event projects onto as many feature maps as there are kernels in the layer.

[0064] According to this embodiment, each kernel produces an output feature map associated with it.

[0065] According to another embodiment of the present invention, each layer of the sCNN further includes a destination mapper, which is connected to an output, in particular a bus of a neuron module, where the destination mapper is configured and arranged to dump outgoing spike events and / or generate outgoing destination information of outgoing spike events received from the neuron module of the current layer, and in particular to associate the destination information with the outgoing spike events, in particular where the outgoing destination information includes information of at least one destination layer comprised by a plurality of layers to which the outgoing spike events are delivered, in particular the destination information includes information of an output feature map where the spike event is generated, a neuron coordinate of the spike event in the output feature map, and / or an offset value for an index of the feature map.

[0066] In this embodiment, each layer can determine destination information autonomously such that sCNN can be extended by adding more layers in a modular manner.

[0067] According to another embodiment of the present invention, the neuron module comprises a plurality of parallel operating neuron sub-interfaces configured to process received neuron addresses and kernel values, and in particular received bias values, and generate updated neuron states and spike events, where each neuron sub-interface comprises an associated neuron memory for reading the neuron state for a received neuron address and writing the updated neuron state to the received neuron address in the associated neuron memory, such that parallel processing of neuron states is achieved by the neuron sub-interfaces and their associated neuron memories.

[0068] Since read and write operations on memory are usually equally slow and time-consuming operations, this embodiment avoids a potential bottleneck in the processing speed of the network and allows for efficient and fast processing of spike events from previous layers. According to this embodiment, the determined neuron addresses and applicable kernel values ​​are distributed to multiple neuron sub-interfaces for processing in a parallel manner.

[0069] According to another embodiment of the present invention, the neuron module comprises a router module configured, adapted and connected to receive the determined neuron address and applicable kernel value, and in particular bias value, particularly from the kernel module and / or bias module, and to transmit the received neuron address and kernel value, and in particular bias value, to a plurality of parallel operating neuron sub-interfaces configured to process the received neuron address and kernel value, and in particular bias value in parallel, wherein the neuron module further comprises a merge module configured and adapted to receive spike events generated from a plurality of neuron sub-interfaces and funnel the spike events for serial processing, particularly on a bus having fewer channels than the neuron sub-interfaces, particularly where the bus has only a single transmission channel.

[0070] This embodiment allows the Neuron module to accommodate serial bus connections to and from the Neuron module while still facilitating parallel processing.

[0071] According to another embodiment of the present invention, each layer further comprises a bias module comprising an associated bias memory and a bias interface particularly connected to the associated bias memory, where the associated bias memory is configured to store bias values ​​at bias addresses, the bias module is connected to the neuron module, and the bias module is configured to provide bias values ​​to the neuron module particularly for predetermined time intervals, and the neuron module is configured to update the neuron states of all neurons in at least one output feature map based on the received bias values.

[0072] The bias interface, similar to the kernel interface, is configured to address memory for receiving and specifically processing the bias values.

[0073] The bias module is not specifically connected to the memory mapper, so the bias memory provides bias values ​​to the neuron modules, and in particular to the at least one output feature map, independently of the information contained in the incoming spike events.

[0074] According to another embodiment of the invention, the neuron module comprises: - receiving neuron addresses and kernel values, and in particular bias values, in particular from a kernel module and / or a bias module, and distributing in particular said neuron addresses, said kernel values ​​and said bias values ​​to one of the neuron sub-interfaces, - reading the neuronal state of the received neuronal address from the specifically associated neuronal memory; - updating, in particular at said neuron sub-interface, the read neuron state with said received kernel values ​​and / or in particular with said received bias values, - comparing said updated neuron state with at least one threshold value, in particular stored and accessible in a threshold register, in particular wherein said register is programmable with at least one threshold value; - generating a spike event, which is provided in particular to a destination mapper via a merge module, when said at least one threshold is reached, and resetting said updated neuron state to a reset neuron state; - writing said updated, i.e. in particular said reset, neuron state to said received neuron address in an associated neuron memory; It is structured as follows.

[0075] An sCNN may include, for example, two thresholds for each neuron: a lower threshold and an upper threshold.

[0076] When a spike event occurs by a neuron, the neuron state is set to a reset neuron state, which may be, for example, a difference between the updated neuron state and a threshold that is equal to or equal to zero.

[0077] The neuron module, and in particular the neuron sub-interface, is therefore adapted to carry out the necessary calculations to update the neuron state of the neuron onto which the incoming spike event projects, in particular with kernel values ​​applicable in a parallel manner, and thus the neuron module comprises the electronic circuitry necessary to carry out such a task.

[0078] According to another embodiment of the present invention, the neuron module comprises a plurality of neuron sub-interfaces having associated neuron memories for parallel access to the determined neuron addresses, wherein the neuron module is configured to distribute the plurality of received neuron addresses to the plurality of neuron sub-interfaces, in particular by a neuron router module, each neuron sub-interface connecting with a sub-block of its associated memory is configured to process the received neuron address and kernel value.

[0079] This embodiment allows for parallel processing of the neurons to be updated with kernel values.

[0080] In particular, reading from and writing to the associated neuron memory are equally slow processes, so parallelizing this process reduces processing time when using traditional memory storage, e.g., random access memory.

[0081] This embodiment allows for real-time processing of incoming spike events, especially at high data rates.

[0082] According to another embodiment of the present invention, the kernel interface comprises a plurality of kernel read / write interfaces for parallel access to sub-blocks of kernel memory contained in an associated kernel memory, wherein said kernel interface is configured to distribute received kernel addresses to the plurality of kernel read / write interfaces, in particular by a kernel interface router system, and to collect kernel values ​​received from the sub-blocks of kernel memory.

[0083] This embodiment solves the problem of accelerating the equally slow read and write operations in traditional memory storage, making sCNN particularly capable of real-time processing, even at high data rates.

[0084] The term "real-time" processing or operation is particularly characterized in that the average processing rate of incoming spike events by the sCNN is equal to the average rate of incoming spike events of the sCNN, such that memory overflow is avoided.

[0085] According to another embodiment of the invention, the memory mapper is configured to perform the following sequence of operations in order to determine a neuron coordinate with a kernel address of an applicable kernel value and a corresponding neuron address to be updated when a spike event is received by the memory mapper, said spike event comprising or consisting of information about the neuron coordinate including the coordinates (x,y) of said spike event in a source feature map, in particular a channel identifier (c), in particular an x ​​identifier and a y identifier: In particular, a given offset (p x ,p y ) the coordinates (x,y) of the spike event, padded with zeros (x+p x ,y+p y )death, - from the coordinates (x,y), in particular the zero-padded coordinates (x+p x ,y+p y) to which the spike event projects, and the corresponding kernel anchor coordinates (x0, y0) of at least one kernel (f) are obtained. k ,y0 k ) where the kernel anchor coordinate (x0 k ,y0 k ) is the kernel coordinate (f,x0 k ,y0 k ), in particular, the neuron anchor coordinates are associated with an output feature map, Starting from the neuron anchor coordinates and the kernel anchor coordinates, find all neuron coordinates (f,x,y) to be updated and all kernel coordinates (c,f,x) of the applicable kernel values. k ,y k ) and in particular, the output feature map size (H, W) provided by the register, the stride size (s x ,s y ), and / or kernel size (H k ,W k ) by processing information in - determining specifically compressed neuron addresses and kernel addresses from the determined neuron coordinates (f,x,y) and the kernel coordinates with the applicable kernel values, - providing said determined neuron address and kernel address to a kernel module and / or a neuron module.

[0086] The source feature map is the output feature map of the layer of sCNN where the spike event was received.

[0087] Padding coordinates from incoming spike events specifically solves the problem of convoluting spike events at the boundaries of feature maps. x ,p y ) is specifically contained in a programmable register.

[0088] A register in the context of this specification refers to a memory device that provides faster reading and writing of stored data, especially compared to memories such as neuron memories or kernel memories. Thus, the sCNN according to the present invention comprises a register for storing information that is accessed particularly frequently.

[0089] The offset value is accessed each time a spike event is received by the layer, so it is stored in a register that is electronically connected to or provided by the memory mapper.

[0090] The neuron anchor coordinates correspond to the neuron located at the corner of the projection window that the spike event projects in the output feature map. The projection window is given by, among other things, the kernel size, e.g., x and y dimensions, the stride size of the convolution, and possibly other factors. The neuron anchor coordinates serve the purpose of defining the starting point in the output feature map for the convolution, and in particular for the sweep operation of sweeping the kernel with the determined neuron address for updating the neuron state of the neuron. The sweep of the kernel in the output feature map depends in particular on the stride size of the convolution.

[0091] Kernel anchor coordinates are the coordinates of kernel values ​​that are specifically located at or near the corners of the kernel. The kernel anchor coordinates serve the purpose of defining a starting point for determining all applicable kernel values. This is especially important when the stride size is greater than 1.

[0092] The kernel anchor coordinates depend, among other things, on the stride size, the kernel size, and possibly other factors.

[0093] Once the neuron anchor coordinates and kernel anchor coordinates have been determined, a starting point is generated for computing the neurons that will be affected by convolution with the applicable kernel values.

[0094] From the starting point defined by the neuron anchor coordinates and the kernel anchor coordinates, all the neuron coordinates (f,x,y) to be updated, i.e., all the neurons onto which the spike event projects, and all the kernel coordinates (c,f,x k ,y k ) is the output feature map size (H,W) provided by the registers, stride size (s x ,s y ), and / or kernel size (H k ,W k ) is determined by processing information in

[0095] The parameters required to determine which neuron coordinates to update are specifically stored in registers that offer fast read and write speeds compared to conventional memory storage.

[0096] From the neuron coordinates, which specifically point to a location in the output feature map, a neuron address is determined, which latter specifically points to a memory address in the neuron memory.

[0097] Once the neuron address and kernel address are determined by the memory mapper, these addresses are provided to the kernel module, specifically the kernel interface, where the applicable kernel values ​​are received and sent to the neuron module, where the neuron state of the determined neuron is updated with the applicable kernel values.

[0098] The memory mapper is configured and arranged to perform all these operations. For this purpose, the memory mapper is specifically configured and arranged to execute a formula that calculates the neuron address to be updated and the kernel address of the applicable kernel value. This formula is specifically built into the memory mapper and is not programmable. However, the values ​​stored in registers to provide, for example, stride size, kernel size, etc. to said formula can be programmed and adjusted by programming the respective registers with the appropriate values.

[0099] According to another embodiment of the present invention, the associated neuron memory is organized such that all neuron states are stored in consecutive neuron addresses, in particular, the associated neuron memory has all neuron states stored in consecutive neuron addresses during operation, the memory mapper is configured to generate consecutive neuron addresses for all neuron coordinates, such as generating compressed neuron addresses, and the memory mapper is configured to provide the compressed neuron addresses to the neuron module.

[0100] This embodiment allows for a memory space efficient use of the associated neuron memory.

[0101] According to another embodiment of the present invention, an associated kernel memory is organized such that all kernel values ​​are stored in consecutive kernel addresses, in particular, said associated kernel memory has all kernel values ​​that are stored in consecutive kernel addresses during operation, said memory mapper is configured to generate consecutive kernel addresses for all kernel coordinates, such as generating compressed kernel addresses, and said memory mapper is configured to provide the compressed kernel addresses to a kernel module.

[0102] This embodiment allows for space-efficient use of the associated kernel memory.

[0103] According to another embodiment of the present invention, an event-driven spiking convolutional network includes multiple layers connected with a spike event router configured to forward outgoing spike events received from a source layer to at least one destination layer, in particular said router having access to destination information associated with the outgoing spike events, said destination layer being possible to be said source layer.

[0104] This embodiment provides an sCNN with multiple layers electronically connected and organized by a spike event router that can be programmed to allow for a variety of sCNN configurations and routing options.

[0105] A dynamic vision sensor configured to generate event-driven spikes can serve as an input or first source layer of a spike event router.

[0106] According to another embodiment of the invention, an event-driven convolutional neural network is constructed and arranged for event-driven, in particular asynchronous processing of spike events, wherein said sCNN is configured to receive asynchronous spike events and in particular process each spike event upon receipt, in particular wherein the memory mapper is an event-driven, in particular asynchronous electronic circuit, in particular the associated memory module is an event-driven, in particular asynchronous module, in particular the kernel module is an event-driven, in particular asynchronous module, in particular the destination mapper is an event-driven, in particular asynchronous electronic circuit.

[0107] According to another embodiment of the present invention, the sCNN is configured to receive and process spike events from a dynamic vision sensor, particularly where said sCNN comprises a dynamic vision sensor.

[0108] The problem according to the invention is further solved by a computer program, which is in particular adapted to operate an sCNN, more in particular a component of an sCNN according to the invention.

[0109] The terms and definitions as introduced in the context of the sCNN embodiment also apply to the computer program.

[0110] The computer program includes instructions for executing the computer program on at least one component of the sCNN, such as a memory mapper, a neuron module, a kernel module, a destination mapper, and / or a spike event router, and when executing the computer program of the event-driven spiking convolutional network according to the present invention causes each component to perform the steps by which the components are configured and arranged to process incoming spike events, particularly asynchronously.

[0111] The computer program specifically provides programmable register values ​​for the registers of the sCNN.

[0112] According to another embodiment of the computer program, the computer program causes the memory mapper to: a) receiving, via an electronic connection, an incoming spike event, in particular a single spike event, from a source layer of the network, the incoming spike event containing information about the coordinates of a particular single neuron in the source layer, and responding to the received spike event; b) determining neuron coordinates and corresponding neuron addresses in associated neuron memories of neurons onto which a convolution of the received incoming spike event with at least one convolution kernel projects; c) determining kernel coordinates with corresponding kernel addresses of kernel values ​​applicable to a neuron state of the determined neuron address, where the neuron state of the determined neuron address is updated with the applicable kernel values, in particular from at least one convolution kernel or from a plurality of convolution kernels; d) providing the determined kernel addresses of the applicable kernel values, and in particular the determined neuron addresses, to a kernel module, in particular a kernel interface; Make them do this.

[0113] According to another embodiment of the computer program, the computer program causes the neuron module to: a) receiving, in particular at one of the neuron sub-interfaces, the determined neuron address and the applicable kernel value; b) for each received neuron address, determining an updated neuron state for the neuron based on an applicable kernel value; c) outputting an outgoing spike event, in particular when, after the neuron state has been updated, the updated neuron reaches, i.e. in particular exceeds or falls below, at least one predefined threshold value; Make them do this.

[0114] According to another embodiment of the computer program, the computer program causes a destination mapper to emit outgoing spike events and / or generate outgoing destination information for outgoing spike events received from a neuron module and associates said destination information with the outgoing spike events, in particular where the outgoing destination information comprises information on at least one destination layer to which the outgoing spike events should be delivered, in particular where the destination information comprises information on an output feature map where the spike event was generated, neuron coordinates of the spike event in the output feature map, and / or an offset value with respect to a feature map index.

[0115] According to another embodiment of the computer program, the computer program comprises the steps of: - receiving neuron addresses and kernel values, and in particular bias values, from a kernel module and / or a bias module, - reading the neuron state of the received neuron address; - updating the read neuron state with the received kernel values ​​and / or in particular with the received bias values; - comparing said updated neuron state with at least one threshold value, in particular stored and accessible in a threshold register, in particular wherein said register is programmable with at least one threshold value; - upon reaching the at least one threshold, generating a spike event that is specifically provided to a destination mapper and resetting the updated neuron state to a reset neuron state; - writing said updated, i.e. in particular said reset, neuron state to said received neuron address in an associated neuron memory; Make them do this.

[0116] According to another embodiment of the computer program, the computer program causes the neuron module to distribute, in particular by a neuron router module, a plurality of received neuron addresses to a plurality of neuron sub-interfaces, wherein the computer program further causes each neuron sub-interface connected to an associated memory to process the received neuron address and kernel value.

[0117] According to another embodiment of the computer program, the computer program causes the kernel interface to distribute kernel addresses received, in particular by the kernel interface router system, to multiple kernel read / write interfaces and to collect kernel values ​​received from sub-blocks of the kernel memory.

[0118] According to another embodiment of the computer program, the computer program causes the memory mapper to execute the following sequence of operations when a spike event is received by the memory mapper to determine neuron coordinates with kernel addresses of applicable kernel values ​​and corresponding neuron addresses to be updated, where the spike event includes or consists of information about the coordinates (x,y) of the spike event in a source feature map and in particular a channel identifier (c), where the neuron coordinates include in particular an x ​​identifier and a y identifier, and the following sequence of operations: In particular, a given offset (p x ,p y) the coordinates (x, y) of the spike event, zero-padded (x+p x ,y+p y )death, - from the coordinates (x,y), in particular the zero-padded coordinates (x+p x ,y+p y ), the neuron anchor coordinates (x0, y0) to which the spike event is projected and the corresponding kernel anchor coordinates (x0 k ,y0 k ) where the kernel anchor coordinate (x0 k ,y0 k ) is the kernel coordinate (f,x0 k ,y0 k ), in particular, the neuron anchor coordinates are associated with an output feature map, Starting from the neuron anchor coordinate and the kernel anchor coordinate, find all neuron coordinates (f,x,y) to be updated (i.e. onto which the spike event projects) and all kernel coordinates (c,f,x) of the applicable kernel values. k ,y k ) and in particular, the output feature map size (H, W) provided by the register, the stride size (s x ,s y ), and / or kernel size (H k ,W k ) by processing information in - determining, from the determined neuron coordinates (f,x,y) and kernel coordinates with the applicable kernel values, specifically compressed neuron addresses and kernel addresses; - providing said determined neuron address and kernel address to a kernel module and / or a neuron module; It is.

[0119] According to another embodiment of the computer program, the computer program organizes the associated neuron memory such that all neuron states are stored in consecutive neuron addresses, wherein the computer program causes the memory mapper to generate consecutive neuron addresses for all neuron coordinates, such as generating compressed neuron addresses, and wherein the computer program further causes the memory mapper to provide the compressed neuron addresses to the neuron module.

[0120] According to another embodiment of the computer program, the computer program organizes the associated kernel memory such that all kernel values ​​are stored in consecutive kernel addresses, wherein the computer program causes a memory mapper to generate consecutive kernel addresses for all kernel coordinates, such as generating compressed kernel addresses, and wherein the computer program further causes the memory mapper to provide the compressed kernel addresses to a kernel module.

[0121] According to another embodiment of the computer program, the computer program causes a spike event router to route an outgoing spike event received from a source layer to at least one destination layer, in particular where the router accesses destination information associated with the outgoing spike event, and where the destination layer may be the source layer.

[0122] The problem according to the invention is further solved by a computer-implemented method, in particular configured to operate an sCNN according to the invention, more particularly a component of an sCNN, further comprising at least some of the features and / or method steps as disclosed for the computer program.

[0123] This problem is further addressed by the memory mapper of the event-driven sCNN.

[0124] The terms and definitions as introduced in the context of the sCNN embodiment also apply to the memory mapper.

[0125] The memory mapper according to the invention is configured to determine kernel addresses and neuron addresses which are updated in an event-driven manner when a spike event is received by said memory mapper, said spike event comprising or consisting of information on the coordinates (x,y) and in particular a channel identifier (c) of the spike event, said coordinates comprising in particular the x- and y-identifiers, said memory mapper performing the following sequence of operations when a spike event is received by said memory mapper: In particular, a given offset (p x ,p y ) and the spike event coordinates (x,y) are padded with zeros (x+p x ,y+p y ) - from the coordinates (x,y), in particular the zero-padded coordinates (x+p x ,y+p y ) to which the spike event projects, and the corresponding kernel anchor coordinates (x0, y0) of at least one corresponding kernel (f) are calculated. k ,y0 k ), where the kernel anchor coordinates (x0 k ,y0 k ) is the kernel coordinate (f,x0 k ,y0 k ), in particular, the neuron anchor coordinates are associated with the output feature map, Starting from the neuron anchor coordinate and the kernel anchor coordinate, find all neuron coordinates (f,x,y) to be updated (i.e., onto which the spike event projects) and all kernel coordinates (c,f,x k ,y k) in particular, the output feature map size (H, W) provided by the registers, the stride size (s x ,s y ), and / or kernel size (H k ,W k ) to make decisions by processing information in - determining, from said determined neuron coordinates (f,x,y) and said kernel coordinates with the applicable kernel values, specifically compressed neuron addresses and kernel addresses; - providing the determined neuron address and kernel address to a kernel module and / or a neuron module for processing the neuron address and kernel address.

[0126] According to another embodiment of the memory mapper, the memory mapper is configured to generate consecutive neuron addresses, such as generating compressed neuron addresses for every neuron coordinate, and wherein the memory mapper is configured to provide the compressed neuron addresses to the neuron module.

[0127] According to another embodiment of the memory mapper, the memory mapper is configured to generate contiguous kernel addresses, such as generating a compressed kernel address for every kernel coordinate, and wherein the memory mapper is configured to provide the compressed kernel addresses to a kernel module.

[0128] In particular, exemplary embodiments are described below in conjunction with figures, which are accompanied by explanatory texts of individual features of the embodiments and aspects of the invention that are attached and illustrated in the claims. Each individual feature illustrated in the figures and / or mentioned in said text of the figures can be incorporated (even separately) in the claims relating to the device according to the invention.

[0129] 1 shows a schematic diagram of one layer 10 of an sCNN 1 according to the present invention. Layer 10 is an electronic module with various components and is typically integrated with multiple copies within an sCNN.

[0130] The boxed areas represent units or modules, and the arrows indicate the electronic data connections between the components and the associated direction of data flow.

[0131] The sCNN 1 comprises a memory mapper 130 configured to receive incoming spike events 140 denoted by {c,x,y}. The incoming spike events 140 are digital data that contain, among other things, information about the location where the incoming spike event 140 was generated and a channel index indicating the channel with which the spike event 140 is associated. The location information is in the form of coordinates in the output feature map of the source layer, with x and y identifiers, or coordinates of a dynamic vision sensor (see FIG. 2). The channel can, for example, denote a color channel from a spike event generating device such as a dynamic vision sensor. However, other channel markers are possible.

[0132] The task of the memory mapper is (currently) to determine all the necessary information for layer 10 to perform the convolution(s). Thus, the memory mapper 130 needs to determine the neuron address 142n to which the incoming spike event 140 projects, i.e. which neuron in said layer is affected by the convolution of the spike event 140 with at least one convolution kernel 410.

[0133] Once the memory mapper 130 receives the spike event 140, the memory mapper 130 begins processing the spike event 140. The memory mapper 130 is an event-driven memory mapper 130, which may include, for example, a processing pipeline with multiple buffer stages and electronic combinatorial logic circuitry that adds, multiplies and multiplexes various bits of the spike events c, x and y.

[0134] The memory mapper 130 is wired to execute a predefined equation configured to determine which neuron address 142n should be updated with the applicable kernel value 142k. As parameters such as the size of the feature map the incoming spike event 140 comes into, the size and number of the output feature map in the current layer 10, the number of kernels, the stride size of the convolution, etc. may vary, these values ​​are provided to the equation by the associated registers or registers 131 provided by the memory mapper 130. All the registers 131, 163, 123 shown in Figure 1 are relatively high speed memory storage devices configured to allow high speed read and write operations.

[0135] It is noted herein that all registers are programmable and specifically configured to store non-trainable or untrained parameters of the sCNN.

[0136] The kernel values ​​142k and bias values ​​are then typically determined during the training procedure of sCNN 1, as is well known in the art.

[0137] Registers 131 connected to the memory mapper 130 store values ​​for the kernel size of at least one kernel, the stride, zero padding of x and y, the number of kernels f, and the number of output feature maps 430.

[0138] Additionally, the register 131 may store values ​​for the size of the output feature map 430, such as its height and width (e.g., measured in pixels).

[0139] An exemplary embodiment of the memory mapper 130 and how the neuron addresses and kernel values ​​are determined by the memory mapper is shown in FIG.

[0140] The determined neuron address n comp and kernel address k comp It has been specially compressed so that no memory space of the associated neurons and kernel memory is wasted.

[0141] The memory mapper 130 outputs the neurons 142n and the kernel addresses to a kernel module 110 that comprises a kernel interface 111 and an associated kernel memory 112. The kernel interface 111 is a kernel memory interface.

[0142] The kernel interface 111 receives the kernel address and reads the kernel address from the associated kernel memory 112 .

[0143] The kernel memory 112 is a conventional memory such as a random access memory (RAM), flash memory, etc. The associated kernel memory 112 may be located on the same circuit board as the rest of the layer 10, or may be connected to the layer 10 to constitute an external memory device.

[0144] Because multiple kernel values ​​142k need to be read from the kernel memory 112 for each spike event 140, and conventional memory devices such as SRAM are relatively slow, the kernel module 110 is constructed and arranged to read and write the kernel values ​​142k in parallel to the associated kernel memory 112 (see FIG. 6).

[0145] An exemplary embodiment that addresses this issue by parallelizing the read and write operations of the associated kernels and / or associated neuron memories 112, 122 is shown in FIG.

[0146] Assuming that the number of convolution kernels in the current layer 10 is N, the kernel interface 111 outputs the kernel weights of the N convolution kernels associated with the neuron addresses, and the neuron addresses will be updated with the values ​​of the N convolution kernels.

[0147] Each convolution kernel 410 (numbered as in FIG. 4) projects onto an associated output feature map 430, so let us assume that the N convolution kernels 410 project onto the N output feature maps 430 of the current layer 10, with kernel dimensions of N×M×H×W, where M is the number of input channels, H is the height of each kernel, and W is the width. Thus, each output feature map 430 is scaled by the applicable kernel value (w k ) for a number of neurons 420 to be updated with the applicable kernel values ​​142k. The neuron addresses of these neurons 420 are provided by the kernel interface 111 in an output data set that includes said neuron addresses 142n associated with the applicable kernel values ​​142k. The neuron addresses of the neurons to be updated are provided by the kernel interface 111, but are calculated by the memory mapper 130 as described above.

[0148] The Kernel module 110 is electronically connected to a Neuron module 120 that is constructed and arranged to process neuronal states of neurons. An output data set from the Kernel module 110 is received by the Neuron module 120. Upon receiving the output data set, the Neuron module 120 begins processing the data set. That is, the Neuron module 120 is event-driven.

[0149] In addition to updating some of the neurons 420 in the output feature map 430, all of the neurons 431 in the output feature map 430 may be updated at repeated time intervals with the bias value 142b.

[0150] To this end, each layer 10, 10', 10'' comprises a bias module 160 having a bias interface 161 and an associated bias memory 162. The associated bias memory 162 is a conventional memory device and is configured to store bias values ​​to be applied to the neurons 431.

[0151] The bias interface 161 is connected to or integrated with a bias-associated memory 162 and configured to read and write bias values ​​from the associated bias memory 162. Furthermore, the bias interface 161 is configured to output the bias values ​​and provide the bias values ​​to the neuron module 120 together with the neuron addresses to be updated with the bias values.

[0152] It is important to note that while the bias value 142b is typically extended to all neurons 431 within a particular output feature map 430, the neurons 420 that should be updated with the kernel value 142k depend on the particular spike event 140 received by the memory mapper 130.

[0153] Thus, the bias module 160 is not connected to the memory mapper 130 to exchange data and / or operate synchronously, but operates independently of and in parallel with any input events being processed. The operation of the bias module can be based on a clocking device, but can also be operated by any trigger selected by the user.

[0154] The bias module 160 comprises a bias register 163, a high speed read / write memory component that stores an output feature map index that indicates for which output feature map 430 in the current layer 10 should be updated with the bias value 142b.

[0155] The neuron module 120 is connected to the bias module 160 and the kernel module 110 and is configured to receive outputs from said modules 160,110.

[0156] The neuron module 120 is connected to a neuron register 123. The neuron module 120 includes a number of neuron sub-interfaces 121s, each having an associated neuron memory 506 (see FIG. 5) that stores the neuron state (see FIG. 6) of the neuron for reading, processing, and writing the neuron state. One embodiment of the neuron sub-interfaces is shown in FIG. 5.

[0157] Each time the neuron module 120 receives an output data set from the kernel module 110 or a bias value 142b from the bias module 160, the neuron module distributes the output data set to one or more neuron sub-interfaces 601 that read the neuron state from the neuron to be updated with the applicable kernel value 142k and / or bias value 142b.

[0158] Because multiple neuron states need to be read from the associated neuron memory 506 according to the spike event 140 or the received bias value 142b, and because conventional memory devices such as SRAM are relatively slow, the neuron module 120 is constructed and arranged to read and write neuron states in parallel to and from the associated neuron memory 506 (see FIG. 6) using parallelized neuron sub-interfaces 121s and their associated neuron memories 506.

[0159] The neuron state is updated with a kernel value of 142k, for example, according to the following formula: s(n+1)=mod(s(n)+wb,tr) where s(n) represents the neuron state of the neuron stored in the associated neuron memory 506, and wb corresponds to either the applicable kernel value 142k or bias value 142b. The state of the new neuron s(n+1) is given, for example, by a modulo operation with upper and lower thresholds tr.

[0160] If s(n)+wb exceeds the threshold, the neuron state is reset to the neuron state reset by the module operation, and an outgoing spike event 150 is generated by the neuron sub-interfaces 121s. The neuron sub-interfaces 121s store the new neuron state in the associated neuron memory 506 under the same neuron address as it was received. This process is also shown in Figure 5 and described in detail. Please note that Figure 5 provides a functional diagram of the neuron module 120, and Figure 6 provides a schematic diagram of the structural organization of the neuron module 120.

[0161] Other ways of defining the reset neuronal state are possible and are detailed in the context of FIG. 5.

[0162] For every neuron that reaches either the lower or upper threshold 502, the neuron interface 601 outputs an outgoing spike event 150, which is a data structure that includes the neuron's coordinates in the output feature map that contains the neuron, and also an output feature map index for indicating the output feature map. This data structure is electronically transmitted to the Destination Mapper 170 (see FIG. 1) component, also called the Destination Mapper in the context of this specification.

[0163] The destination mapper may be, for example, a merge buffer for merging information received from registers 171 and outgoing spike events 150 connected to the destination mapper.

[0164] The destination mapper 170 is constructed and arranged to add information of the outgoing spike event 150, 503 so that it can be routed to the appropriate layer 10, 10', 10'', also referred to as the destination layer in the context of this specification.

[0165] For this purpose, the destination mapper 170 comprises an associated register 171 providing information regarding the destination layer 10, 10', 10" to which the outgoing spike event 150, 503 is to be routed.

[0166] The destination mapper 170 adds information of the outgoing spike event 150 to include the x, y coordinates and index of the output feature map 430 where the spiking neuron is located. Furthermore, the index of the output feature map is included in the additional information indicating the output feature map where the spiking neuron is located. Furthermore, the outgoing spike event 150 can be routed to multiple destination layers 10', 10''. Therefore, the destination mapper 170 adds information to include the index of the destination layer indicating another layer 10', 10'' provided by the sCNN 1 and an offset value for the feature map index so that the outgoing spike event is mapped to a predetermined output feature map in the destination layer. Adding the offset value enables layer stacking in the sCNN 1.

[0167] The outgoing spike events 150, enriched with information as described above, can then be provided to the spike event router 200 of the sCNN, as depicted in FIG.

[0168] FIG. 2 shows the general layout of an sCNN 1 according to the invention, which comprises a spike event router 200 and several layers 10, 10', 10'' (as described in FIG. 1).

[0169] The spike event router 200 receives spike events 210 from layers 10, 10', 10'', or from an event generator 220, such as a dynamic vision sensor.

[0170] The spike event router 200 then provides 230 the spike event 210 for further processing based on the added destination layer 10', 10'' information of the spike event 210.

[0171] The spike event router 200 is, for example, a stream multiplexer / demultiplexer circuit.

[0172] The spike event router 200 is programmable and configured to route spike events to destination layers in a backward, forward, or recurrent manner, providing sCNN 1 with great flexibility, especially with respect to the requirements of deep learning networks.

[0173] The spike event router 200 is configured to provide data flow between layers and to facilitate chip-to-chip communication (denoted as "chip-out" and "chip-in") when multiple sCNNs 1 are arranged in coordination as a network.

[0174] For this purpose, the spike event router 200 comprises an input port 240 for receiving signals and data between chips from an external device (which may be a spike event router from a second sCNN). The spike event router 200 is also configured to receive data, i.e. spike events, generated by a dynamic vision sensor 220 connected to the spike event router 200. The dynamic vision sensor 220 can be treated as a source layer of the sCNN 1, except that recurrent or backward propagation is not possible.

[0175] The spike event router 200 has an output port 250 for outputting spike events to other chips or as a result.

[0176] In particular, sCNN 1 comprises over 100,000 programmable neurons that are assigned to layers 10, 10', 10'' of sCNN 1 and output feature map 430.

[0177] The layers 10, 10', 10'', and in particular the registers 131, 163, 123, 171 of said layers 10, 10', 10'', are appropriately programmed to reflect the particular layout of each layer 10, 10', 10'', i.e., the number of convolution kernels contained in each layer, the dimensions of the output feature maps of each layer, etc.

[0178] The dynamic vision sensor 220 may be, for example, a device having an array of light-receptive pixels, each configured to output a spike event when the intensity of the detected light changes, i.e., the pixels of the dynamic vision sensor 220 are sensitive to changes in the detected light flux.

[0179] The dynamic vision sensor 220 is an event-driven device, i.e., the pixels output spike events as they occur, as opposed to frame-based camera readouts. Such dynamic vision sensors offer the advantage of very fast dynamic imaging. When combined with the sCNN 1 according to the present invention, the potential of real-time ultra-fast data processing can be maximized by utilizing the sCNN 1.

[0180] Figure 3 shows a schematic of the data flow within the memory mapper 130. The purpose of the memory mapper 130 is to determine the neuron 420 to which an incoming spike event 140 projects. This is also shown in Figure 4.

[0181] An incoming spike event 140 is processed by the memory mapper 130 as follows:

[0182] The incoming spike event 140 has, for example, information of the channel and coordinates {c,x,y}. First, zero padding 300 of the received incoming spike event 140 is performed, where the coordinates of the spike event 140 are padded with an offset value p x ,p y Converted by: {c,x,y}→{c,x+p x ,y+p y}

[0183] The zero padding 300 serves the purpose of avoiding edge effects when coordinates are too close to the boundaries of the feature map 430.

[0184] The offset value is provided by a register 300r, 131 associated with the memory mapper 130.

[0185] In the next step, kernel addresses of applicable kernel values ​​of at least one kernel are determined by a module called Kernel Mapper 301. The Kernel Mapper 301 is configured to calculate neuron anchor coordinates of each feature map 430 onto which the incoming spike event 140 projects.

[0186] The neuron anchor coordinates are provided by, for example, the neuron located in the lower right corner of the array in the respective output feature map 430 onto which the event projects. This step includes processing the output feature map size, the convolution kernel size, and the convolution stride. In addition, the kernel anchor coordinates corresponding to the neuron anchor coordinates are calculated. The neuron anchor coordinates and the kernel anchor coordinates serve as the starting point for determining all neurons 420 in the output feature map 430 that should be updated and all applicable kernel values ​​142k that need to be received from the kernel memory 112.

[0187] Obviously, the starting point based on the two anchor coordinates does not have to be the neuron coordinates of the neuron in the lower right corner, but can be any neuron as long as its relationship with the projection of the spike event is known.

[0188] The neuron anchor coordinates and the kernel anchor coordinates are then sent to the address sweep module 302, which is configured to calculate the remaining neuron coordinates of the neurons 420 to which the spike events 140 project and their associated neuron addresses 142n.

[0189] Additionally, all kernel addresses of the applicable kernel values ​​142k are determined by the sweep module 302.

[0190] The computation of neuron coordinates 422 and applicable kernel coordinates 411 is performed by "sweeping" or shifting the convolution kernel 410 over the output feature map 430 according to the stride size and the determined anchor coordinates, for example as shown in FIG. 4.

[0191] From the neuron coordinates 422 and the kernel coordinates 411, the neuron address 142n and the kernel address are determined by the sweep module 302. To this end, the sweep module may access the programmable register 302r.

[0192] The neuron addresses and kernel addresses are compressed as described above so that memory space is optimally conserved in the neuron memory 506 and the kernel memory 112 .

[0193] The compressed neuron addresses and kernel addresses are then provided from the memory mapper 130 to the kernel module 110 as described above.

[0194] FIG. 4 shows a schematic of the procedure for determining the neuron address onto which an incoming spike event 140 projects and the applicable kernel value 142k.

[0195] The spike event 140 comes from a source feature map 440, specifically an output feature map 430 in the source layer, which is shown in the left column of FIG. 4. Each square in the left column represents a neuron with associated coordinates {x,y} in the source layer feature map 440, SFM, where the neuron 421 from which the spike event 140 originates is colored black. For example, looking at panel A of FIG. 4, the current layer 10, where the spike event 140 is received, includes two kernels 410K1, K2, each with a kernel size (H k ,W k ). The current layer 10 therefore also includes two output feature maps 430 OFM1, OFM2. In this example, the size (H,W) of the output feature maps OFM1, OFM2 is 3x3 neurons each (each neuron is shown as a square). Panels B-H of FIG. 4 show the same situation at different processing steps of the convolution.

[0196] Stride s x =s y are set in two directions, x and y, that is, the convolution kernels K1, K2 are shifted by two steps on the output feature maps OFM1 and OFM2.

[0197] The neuron anchor coordinates x0, y0 and the kernel anchor coordinates corresponding to the first kernel K1 are determined such that the neuron anchor coordinates x0, y0 are located at the lower right corner of the projection portion projected by the spike event 140 / 421, as shown in panel A of Fig. 4. Then, the sweep module sweeps the stride size s xto determine another kernel coordinate and another neuron coordinate (shown as neuron 421) onto which the spike event 140 projects. The sweep is illustrated by a boxed region (and always neuron 421) containing nine neurons of the source feature map 440 SFM.

[0198] In the next step (Panel C), a first convolution kernel K1 is swept along the y direction (with stride 2) to determine the neuron coordinates 422 and kernel coordinates of the first output feature map OFM1.

[0199] In panel D of FIG. 4, the first convolution kernel K1 is shifted along x to determine the last of the four neuron coordinates of the first output feature map OFM1 and the kernel coordinates of the first output feature map OFM1.

[0200] Then, the same procedure (compare panels G and H) is performed for the second kernel K2. In this way, a total of eight neuron coordinates and eight kernel coordinates are determined by the sweep module from the two kernels K1, K2 in the two output feature maps OFM1 and OFM2.

[0201] For the determined neuron coordinates 422 and kernel coordinates 411, corresponding (8) neuron addresses 142n and (8) kernel addresses with applicable kernel values ​​142k are determined so that the convolution is performed by the neuron module 120.

[0202] FIG. 5 shows a detailed schematic of how the convolution is performed on the Neuron module 120, and in particular on the Neuron sub-interfaces 121s.

[0203] When a neuron sub-interface receives a neuron address 142n and an applicable kernel value 142k, for example from the neuron router module 601, the neuron sub-interface(s) 121s reads (500) the neuron state stored under the received neuron address 142n from the associated neuron memory 506. The applicable kernel value 142k is added (501) to the neuron state. The resulting neuron state is then compared to at least one threshold 502, typically two thresholds, namely a lower and an upper threshold 502.

[0204] If the resulting neuron state exceeds one of the thresholds 502 (either above an upper threshold or below a lower threshold), an outgoing spike event 150 is generated by the neuron sub-interface(s) 121s and the neuron state is reset to a reset neuron state, which in this example can be either zero or the remainder of the modulo operation.

[0205] The reset neuron state is written back to the associated neuron memory 506 of the neuron sub-interface(s) 121s (505). If no spike event is generated, the resulting neuron state is written back to the associated neuron memory 506 (505).

[0206] 5 is applied in the same manner to the bias values ​​142b and corresponding neuron addresses received at the neuron sub-interfaces 121s, where instead of the applicable kernel values ​​142k, the bias values ​​142b are added to the neuron state. The remaining processing is handled in the same manner as described for the reception of the kernel values ​​142k.

[0207] FIG. 6 shows in a schematic way how the parallel access and pipeline processing of the associated kernels 112, or neuron memories 506, is facilitated. In FIG. 6, it is shown that the neuron module 120 is configured to address a number of neuron sub-interfaces 121s, where each neuron sub-interface 121 is configured to read, write and process neuron addresses and neuron states as described above. For this purpose, the neuron module 120 comprises a neuron router module 601 that receives the applicable kernel values ​​and the determined neuron addresses to be updated. The neuron module may also receive bias values ​​for distribution to the determined neuron addresses. The router module 601 transmits the received kernel values ​​and the applicable neuron addresses to the corresponding neuron sub-interfaces 121s of the multiple parallel organized neuron sub-interfaces 121. In the neuron sub-interfaces 121s, the applicable neuron addresses are read from the associated memory 506 and updated with the kernel values ​​142k, in particular the bias values ​​142 (see FIG. 5). Once the neuron state of the determined neuron address is updated, the updated neuron state is written back, i.e., stored in the associated memory 506. Depending on the updated neuron state, the neuron sub-interface may generate and output spike events. Since the neuron interfaces are configured to operate in parallel, the relatively slow read and write process in the neuron memory is compensated for so that the processing speed is maintained. The neuron module further comprises a merge module 602 that merges the spike events generated from multiple neuron sub-interfaces 121s for further processing on a common, particularly serial, connection.The neuron module 120, with each neuron sub-interface 121s having its own accessible associated memory 506, allows for effectively parallelized processing of reading, writing, and updating multiple neuron states such that the processing speed of the layer 10 is increased compared to non-parallel processing.

[0208] A similar architecture may be implemented in kernel module 110, such that reading and writing of kernel values ​​in said kernel module is parallelized accordingly.

[0209] Thus, the kernel module 100 comprises a plurality of kernel sub-interfaces each having an associated kernel memory for reading and writing kernel values. The plurality of kernel sub-interfaces are connected to a kernel router module configured to distribute kernel addresses to the kernel sub-interfaces such that kernel values ​​associated with the kernel addresses are read from the associated kernel memory. Additionally, the kernel module may comprise a kernel merge module configured to funnel kernel values ​​provided from the plurality of kernel sub-interfaces to a serial bus.

[0210] The spiking convolutional neural network according to the present invention provides dedicated electronic circuitry for operating modern spiking convolutional neural networks in a memory and energy efficient manner. [Explanation of symbols]

[0211] List of codes 1. Event-driven spiking convolutional neural network 10, 10', 10'' layers 110 Kernel Modules 111 Kernel Interface 112 Kernel Memory 120 Neuron Module 121s Neuron Subinterface 123 Registers of the Neuron Module 130 Memory Mapper 131 Memory Mapper Registers 140 Incoming Spike Events 141 Kernel Address and Neuron Address 142b Bias value 142n Address(es) of the neuron(s) to be updated 142k applicable kernel value 150 outgoing spikes 160 Bias Module 161 Bias Interface 162 Bias Memory 163 Bias Module Registers 170 Destination Mapper 171 Destination Mapper Register 200 Event Router 210 Outgoing Spike Event 220 DVS sensor 230 Iriki Spike Event 240 inputs / chip-to-chip input ports 250 outputs / chip-to-chip output port 300 Zero Padding Modules / Movement 300r Associated Registers 301 Kernel Mapping Modules / Operations 302 Address Sweep Module / Operation 302r Associated Register 410 Convolution Kernels 411 Kernel coordinates, K1,K2 420 Decided / Updated Neurons 421 Source layer neuron coordinates 422 Neuron coordinates of the determined neurons 430 Output feature maps, OFM1, OFM2 431 neurons in the output feature map 440,SFM Source Feature Map Read 500 neuron states 501 Update neuron state 502 Threshold 504 Threshold register 505 Write neuron state 601 Neuron Router Module 602 Neuron Merge Module

Claims

1. 1. An electronic circuit constructed and arranged to operate an event-driven spiking convolutional neural network (1), the electronic circuit comprising a plurality of layers, each layer comprising: a kernel module (110) adapted to store and process kernel values ​​of at least one convolution kernel (410) in an event-driven manner; a neuron module (120) configured to store and update neuron states of neurons of said event-driven spiking convolutional neural network (1) in an event-driven manner and to output spike events (150) generated from the updated neurons (420); a destination mapper (170) configured to generate outgoing destination information for outgoing spike events (150) received from said neuron modules (120) of the current layer and to add said outgoing destination information to said outgoing spike events (150); a memory mapper (130) configured to determine a neuron (420) to which an incoming spike event (140) from a source layer projects by convolution with at least one convolution kernel (410), such that a neuron state of said determined neuron (420) is updated with an applicable kernel value of said at least one convolution kernel (410), said memory mapper (130) being configured to process the incoming spike event in an event-driven manner; The electronic circuit.

2. 2. The electronic circuit of claim 1, wherein the kernel module (110) comprises a kernel interface (111) connected to an associated kernel memory (112), the associated kernel memory (112) configured to store kernel values ​​of the at least one convolution kernel at a kernel address, and the kernel module (110) is electronically connected to the neuron module (120).

3. The memory mapper (130) is electronically connected to the kernel module (110), the memory mapper (130) comprising: a) receiving an incoming spike event (140) from a source layer of the event-driven spiking convolutional neural network (1), the incoming spike event (140) including information of neuron coordinates (421) in the source layer; and in response to the received spike event (140): b) determining neuronal coordinates (422) and corresponding neuronal addresses of neurons (420) onto which a convolution of the received spike event (140) with the at least one convolution kernel (110) projects; c) determining a kernel address using the corresponding kernel coordinates (411) of the kernel values ​​applicable to the neuron state of the determined neuron address, where the neuron state of the determined neuron address is updated with the applicable kernel values; d) providing the determined neuron address to the kernel module; 13. The electronic circuit of claim 1 constructed and arranged.

4. 4. The electronic circuit of claim 3, wherein the kernel module (110) is configured and arranged to receive the determined kernel address and provide the applicable kernel value stored at the determined kernel address together with the determined neuron address to the neuron module (120).

5. The neuron module (120): a) receiving the determined neuron addresses (142n) and the applicable kernel values ​​(142k); b) for each received neuron address (142n), determining an updated neuron state for the neuron (420) based on the applicable kernel value (142k); c) outputting outgoing spike events (150) of the updated neurons (420) that reach at least one predetermined threshold (502); 4. The electronic circuit of claim 3 constructed and arranged.

6. 2. The electronic circuit of claim 1, wherein the memory mapper is configured to determine, for each received spike event, at least one output feature map of neurons to be assigned to the output feature map, the number of output feature maps being equal to the number of convolution kernels of the current layer.

7. 2. The electronic circuit of claim 1, wherein each layer further comprises a bias module having an associated bias memory and a bias interface connected to the associated bias memory, the associated bias memory being configured to store bias values ​​at bias addresses, the bias module being connected to the neuron module and configured to provide bias values ​​to the neuron module at predetermined time intervals, the neuron module being configured to update neuron states of all neurons in at least one output feature map based on the received bias values.

8. 2. The electronic circuit of claim 1, wherein the neuron module (120) comprises a plurality of parallel-operating neuron sub-interfaces (121s) configured to process received neuron addresses (142n) and kernel values ​​(142k), and received bias values ​​(142b), and to generate (501, 502) updated neuron states and spike events (150), each neuron sub-interface (121s) having an associated neuron memory (506) for reading the neuron states of the received neuron addresses and for writing the updated neuron states to the received neuron addresses in the associated neuron memory (506), such that parallel processing of neuron states is achieved by the neuron sub-interfaces and their associated neuron memories (506).

9. the neuron module (120) comprises a router module (601) configured and adapted to receive determined neuron addresses (142n), kernel values ​​(142k), and bias values ​​(142b) from the kernel module (110) and / or bias module (160) and to transmit the received neuron addresses (142n), kernel values ​​(142k), and bias values ​​(142b) to the plurality of parallel-operating neuron sub-interfaces (121s) configured to process the received neuron addresses (142n), kernel values ​​(142k), and bias values ​​(142b) in parallel, 9. The electronic circuit of claim 8, further comprising a merge module (602) configured and adapted to receive spike events (150) generated from the multiple parallel-operating neuron sub-interfaces (121s) and send the spike events for serial processing to a bus having fewer channels than the neuron sub-interfaces, wherein the bus has only a single transmission channel.

10. The neuron module (120): receiving, in a router module (601), neuron addresses (142n) and kernel values ​​(142k) and bias values ​​(142b) from said kernel module (110) and / or bias module (160); - reading (500) the neuron state of the received neuron address (142n) from the associated neuron memory (506) in one of the parallel-operating neuron sub-interfaces (121s); updating (501) the read neuron states on the parallel-operating neuron sub-interfaces (121s) with the received kernel values ​​(142k) and / or the received bias values ​​(142b); comparing (502) said updated neuron state with at least one threshold value stored in and accessible from a threshold register on said parallel-operating neuron sub-interface(s) (121s); if said at least one threshold is reached (502), generating (503) a spike event that is provided to a destination mapper (170) and resetting said updated neuron state to a reset neuron state; - writing (505) said updated neuron state to said received neuron address (142n) in said associated neuron memory (506); The electronic circuit according to claim 9 , configured as follows:

11. The memory mapper (130) is configured to perform the following sequence of operations to determine the kernel addresses of the applicable kernel values ​​(142k) and neuron coordinates (421) having corresponding neuron addresses (142n) to be updated when a spike event (140) is received by the memory mapper (130), where the spike event (140) includes or consists of information regarding the coordinates (x, y) of the neuron (421) that generated the spike event (140) in a source feature map (440) and the coordinates (x, y) including a channel identifier (c), an x ​​identifier, and a y identifier, the following sequence of operations: - a given offset (p ｘ , p ｙ ) the coordinates (x, y) of the spike event (140) with zero padding (x+p ｘ , y+p ｙ )death, - from the coordinates (x, y) to the zero-padded coordinates (x+p ｘ , y+p ｙ ) to which the spike event is projected, the neuron anchor coordinate (x ０ , y ０ ), and the corresponding kernel anchor coordinates (x ０ ｋ , y ０ ｋ ) where the kernel anchor coordinates (x ０ ｋ , y ０ ｋ ) is the kernel coordinates (f, x ) of each of the at least one kernel (f). ０ ｋ , y ０ ｋ ), and the neuron anchor coordinates (x ０ , y ０ ) is associated with the output feature map (430), Starting from the neuron anchor coordinates and the kernel anchor coordinates, all neuron coordinates (f,x,y) to be updated and all kernel coordinates (c,f,x) of the applicable kernel values ​​(142k) are ｋ , y ｋ ) by processing information on the size, stride size and / or kernel size of the output feature map (430) provided by the registers; - all the determined neuron coordinates (f,x,y) and all the kernel coordinates (c,f,x) of the applicable kernel values ​​(142k) ｋ , y ｋ ) determining a compressed neuron address and a compressed kernel address; providing said determined compressed neuron addresses and compressed kernel addresses to said kernel module (110) and / or to said neuron module (120); 4. The electronic circuit of claim 3 ,

12. 9. The electronic circuit of claim 8, wherein the associated neuron memory (506) is organized such that all neuron states are stored in consecutive neuron addresses, the memory mapper (130) is configured to generate for all neuron coordinates (f, x, y) consecutive neuron addresses, such as generating compressed neuron addresses, and the memory mapper (130) is configured to provide the compressed neuron addresses to the neuron module (120).

13. 2. The electronic circuit of claim 1, wherein the event-driven spiking convolutional neural network (1) comprises a plurality of layers (10, 10', 10'') connected with a spike event router (200) configured to route outgoing spike events received from a source layer to at least one destination layer, the spike event router (200) having access to destination information associated with the outgoing spike events (150), where the destination layer (10'') can be the source layer (10').

14. 14. The electronic circuit of claim 1, wherein the layer refers to at least one electronic circuit representing a layer in the spiking convolutional neural network, the event-driven spiking convolutional neural network operating without a frame or clock.

15. The electronic circuit of any one of claims 1 to 14, wherein the memory mapper (130), the neuron module (120) and the kernel module (110) have hardwired communication in the electronic circuit.

16. The electronic circuit of any one of claims 1 to 14, wherein the memory mapper (130) is an asynchronous electronic circuit.

17. The electronic circuit of claim 3 , wherein the memory mapper (130) is wired to execute a predefined formula configured to determine the neuron addresses to be updated by applicable kernel values.

18. 20. The electronic circuit of claim 17, wherein the memory mapper (130) is capable of providing programmable register values ​​that specify the formula for the determination.

19. 17. The electronic circuit of claim 16, wherein the memory mapper (130) is an asynchronous serial interface circuit with a parallel interface of a predetermined bandwidth so as to be able to process at least one spike event at a time.

20. 17. The electronic circuit of claim 16, wherein the spiking convolutional neural network is configured to receive asynchronous spike events and to process each spike event as it is received.

21. 21. The electronic circuit of claim 20, wherein the kernel module (110) is an asynchronous module and the destination mapper (170) is an asynchronous electronic circuit.

22. The electronic circuit of any one of claims 1 to 14, wherein the multiple layers are electronically connected and organized by a spike event router that can be programmed to enable various spiking convolutional neural network configurations and routing options.

23. 23. The electronic circuit of claim 22, wherein the spiking convolutional neural network is configured to receive and process spike events from a dynamic vision sensor, the dynamic vision sensor configured to generate event-driven spikes can serve as an input or first source layer of the spike event router.

24. The electronic circuit of claim 2 , wherein the associated kernel memory (112) is physically provided by each layer.

25. 9. The electronic circuit of claim 8, wherein each associated neuron memory (506) is a memory storage physically located in each layer or is a memory that is assigned to and connected to said layer, but is not provided in said layer.

26. The electronic circuit of any one of claims 1 to 14, wherein the memory mapper (130) starts processing the incoming spike event (140) when the incoming spike event (140) is received by the memory mapper (130), the memory mapper (130) being event driven and comprising a processing pipeline with multiple buffer stages and electronic combinatorial logic circuits for adding, multiplying and multiplexing various bits of the spike event c, x, y, the incoming spike event (140) including information about coordinates (x, y) and a channel identifier (c).

27. The neuron anchor coordinates (x ｏ , y ｏ ) is given by a neuron located in the lower right corner of the array in each output feature map (430) that the event projects, The neuron anchor coordinates (x ｏ , y ｏ ) ０ ｋ , y ０ ｋ ) is calculated, The neuron anchor coordinates (x ｏ , y ｏ ) and the kernel anchor coordinates (x ０ ｋ , y ０ ｋ 12. The electronic circuit of claim 11, wherein the input spike event (140) is then sent to an address sweep module (302) configured to calculate the remaining neuron coordinates and their associated neuron addresses (142n) of the neuron (420) onto which the incoming spike event (140) is projected.

28. An electronic circuit according to any preceding claim, wherein the registers of the layers are programmed to reflect a particular layout for each layer.

29. 8. The electronic circuit of claim 7, wherein the bias module (160) comprises a bias register (161) that stores an output feature map index that indicates for which output feature map in a current layer should be updated with a bias value.

30. The electronic circuit of any one of claims 1 to 14, wherein the destination mapper (170) is an event-driven, asynchronous electronic circuit.

Images