Method, apparatus and computer program for creating a spiking neural network

Abstract

The present invention relates to a method (10) for creating a spiking neural network. The method begins by assigning a predeterminable control pattern (rollout pattern) to a deep neural network. The deep neural network is then taught using the control pattern. The deep neural network is then converted into the spiking neural network, wherein delays are respectively assigned to the connections of the spiking neural network according to the control pattern. The present invention also relates to a computer program and a device for implementing the method (10) and a machine-readable storage element on which the computer program is stored.

Description

Technical Field

[0001] The present invention relates to a method for creating a spiking neural network by converting a taught artificial neural network into a spiking neural network. The invention also relates to a device and a computer program each configured to carry out the method. Background Art

[0002] It is possible to run artificial neural networks fully in parallel, as shown by the authors Volker Fischer, Jan Köhler, and Thomas Pfeil in their publication “The streaming rollout of deep networks—towards fully model-parallel execution.” arXiv preprint arXiv:1806.04965 (2018) and in DE 20 2018 104 373 U1.

[0003] Spiking Neural Network (English S piking N eural N SNN is an artificial neural network ( A rtifical N eural N A spiking neural network is a variant of artificial neural networks (ANNs) and closely resembles biological neural networks. As in biological neural networks, neurons in spiking neural networks do not fire during every propagation cycle, as in deep neural networks. Instead, they fire only when the membrane potential exceeds a threshold. When a spiking neural network neuron fires, it generates a short pulse, which is transmitted to other neurons, which in turn raise or lower their membrane potential based on the pulse.

[0004] Spiking neural networks are difficult to teach because short spike trains are represented by Dirac functions, which cannot be derived mathematically.

[0005] It is possible to convert a trained artificial neural network into a trained spiking neural network, as shown by the authors Rueckauer Bodo, Lungu Iulia-Alexandra, Hu Yuhuang, Pfeiffer Michael, and Liu Shih-Chii in their publication “Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification” Frontiers in Neuroscience doi 10.3389 / fnins.2017.00682 (2017), https: / / doi.org / 10.3389 / fnins.2017.00682.

[0006] Advantages of the present invention

[0007] Spiking neural networks are highly efficient during inference because their layers, particularly neurons, can be executed completely in parallel on dedicated hardware. However, this advantage cannot be fully exploited when converting artificial neural networks, particularly those with skip / recurrent connections, into spiking neural networks. This is because, if an artificial neural network with skip connections is converted, the spiking neural network can only be executed sequentially, for example by waiting for a clock, which is inefficient.

[0008] Artificial neural networks also differ from spiking neural networks in that they do not integrate information over time. Instead, they only process and then forward the currently available information in each transmission cycle. This means that artificial neural networks operate sequentially, while spiking neural networks operate in parallel.

[0009] This can result in information in the spiking neural network created by converting the artificial neural network not being available at the correct neuron at the correct time. Furthermore, there is the problem that the temporal integration of the spiking neural network is not taken into account when teaching the artificial neural network. This can have a negative impact on the accuracy and reliability of the results of the spiking neural network.

[0010] In contrast, the method presented below has the following advantages: The operation of spiking neural networks is already taken into account when teaching the deep neural network. Furthermore, the temporal integration of information from the spiking neural network can already be taken into account when teaching the deep neural network. Consequently, the subsequent conversion of the deep neural network into a spiking neural network results in a particularly efficient spiking neural network with high accuracy. Summary of the Invention

[0011] In the first aspect, a method for creating a spiking neural network (SNN) is introduced. S piking N eural N A method, particularly a computer-implemented method, is provided for a deep neural network (SNN). The method comprises the following steps: assigning a predefinable control pattern (rollout pattern) to a deep neural network. The deep neural network comprises multiple layers, each of which is connected to one another in a predefinable order. The control pattern characterizes at least one order of calculations, particularly sequential calculations, in which layers or neurons of the deep neural network determine their intermediate variables. The control pattern may also characterize multiple orders, which can then be executed in parallel during operation of the deep neural network according to the control pattern. The control pattern further characterizes that at least one of the layers of the deep neural network determines its intermediate variables independently of the order. Preferably, using (non-)linear activation functions, the neurons or layers determine their intermediate variables based on the input variables supplied to them. The neurons or layers output their intermediate variables, which in turn serve as input variables for subsequent neurons / layers. The deep neural network is then taught using the control pattern, particularly using training data. That is, during the teaching process, the training data can be propagated through the deep neural network according to the control pattern. The deep neural network is then converted into a spiking neural network. According to the control mode, a time delay is assigned to each connection or neuron of the spiking neural network. It can be said that the delay corresponds to the physical duration of the delayed output of a short pulse or a short pulse sequence by a neuron of the spiking neural network, or the delayed forwarding of the sequence by the connection, or the delayed processing of the short pulse / pulse sequence at the target neuron. The time delay can be assigned to the following connection of the spiking neural network, which corresponds to the corresponding connection of the deep neural network, which connects the layer or neuron of the deep neural network that determines the intermediate variable independently of the order with the subsequent layer / neuron. The time delay can be assigned to the following neuron of the spiking neural network, which corresponds to the corresponding neuron of the deep neural network, which determines the intermediate variable independently of the order.

[0012] It is conceivable that the spiking neural network is subsequently stored in a memory or that the spiking neural network is put into operation. Operation of a spiking neural network can be understood as: the spiking neural network receives input variables, which are transmitted / processed by the spiking neural network according to an assigned delay, and the spiking neural network outputs output variables, such as classification, regression, etc. The delay may result in the connections of the spiking neural network forwarding short pulses with a time delay, or alternatively, the layers / neurons of the spiking neural network outputting their short pulses with a delay.

[0013] The sequence can define the order in which the layers of the deep neural network determine their intermediate variables / output variables. For example, the sequence can correspond to the order in which the layers are arranged in the deep neural network. After the "conventional / sequential" operation of the deep neural network, each layer gradually determines its intermediate variables according to its position in the sequence, and during this time all other layers are inactive. If the control mode indicates that one of the layers determines its intermediate variables independently of the sequence, then this layer is active, regardless of its position in the sequence.

[0014] A spiking neural network can be understood as an artificial neural network in which its neurons output short pulses (spikes). An artificial neural network can be understood as a plurality of interconnected layers inspired by biological neural networks.

[0015] Artificial neural networks are based on a collection of connected neurons that model neurons in a biological brain. Each connection, like a synapse in a biological brain, can transmit an intermediate variable from one artificial neuron to another. These layers, and in particular neurons, are connected to each other via connections. These connections forward the intermediate variables of a layer / neuron and provide these intermediate variables as input variables for subsequent connected layers / neurons. These connections can each be assigned a weight that weights the intermediate variables. The artificial neuron that receives the intermediate variable can process the intermediate variable and then forward it to other artificial neurons connected to it via its connections. In deep neural networks, the intermediate variables can be real numbers, and the output of each artificial neuron is calculated as a (non-)linear function of the sum of its inputs. In contrast, in a spiking neural network, the rate at which short pulses are transmitted between neurons can essentially correspond to this real number on average.

[0016] It has been recognized that the teaching of a deep neural network with the sequential propagation of training data through the deep neural network is completely contrary to the completely parallel implementation mode of a spiking neural network. This is because in a spiking neural network, all neurons can update their states simultaneously. Therefore, the conversion of a deep neural network that has been taught sequentially can lead to a spiking neural network in which information is processed at the wrong point in time during the propagation process. The advantage of this method is that the control mode can ensure that the information is present at the correct corresponding neuron of the spiking neural network at the correct point in time. The adaptation of the delays of the connections of the spiking neural network according to this control mode allows the temporal adjustment of this information during the propagation through the spiking neural network. Therefore, the proposed method leads to a spiking neural network with better performance, because the parallel implementation of the layers / neurons is taken into account when teaching the deep neural network.

[0017] Another advantage is that, due to the improved teaching of deep neural networks (using this control mode), the conversion or transformation of deep neural networks results in spiking neural networks that can be better managed over time. Temporal integration is understood to mean that the information accumulated in the neurons can be retained within a predeterminable time window and combined with the newly added information within this time window. This allows for better utilization of information over time. This is reflected in the accuracy of the determined output variables of the spiking neural network.

[0018] It can be said that when converting a deep neural network into a spiking neural network, another neural network with the same structure as the deep neural network is created, which satisfies the characteristics of the spiking neural network. During this conversion, each neuron of the deep neural network can be replaced by a neuron of the spiking neural network. It is preferably noted that the firing rate of the neurons of the spiking neural network corresponds, on average, to the activation of the corresponding neurons of the deep neural network for predefined input variables. The ReLu activation function is advantageous for deep neural networks because it enables the use of robust normalization techniques that linearly scale all weights of a layer to achieve a sufficiently high firing rate in the spiking neural network to maintain activity without reaching saturation.

[0019] It is proposed that, when assigning the control mode, a control variable be assigned to each connection or each layer and / or each neuron, which control variable characterizes whether the intermediate variables of the corresponding subsequently connected layers / neurons are determined in accordance with the order or independently of the order. A control variable is assigned to at least one of the layers so that the intermediate variables of the layer are determined independently of the order. Independent of the order can be understood as meaning that the calculation of the intermediate variables of the layers is performed decoupled from the order.

[0020] It is proposed that when controlling the calculation of the deep neural network according to the control mode, each of the layers determines its intermediate variables step by step, in particular sequentially, according to the sequence of the control mode, in particular at a predefinable simulation time of a sequence of simulation time points. The sequence of simulation time points can be adapted to a physical time grid or correspond to it. The layers whose intermediate variables are determined independently of the sequence each determine their intermediate variables at each step, in particular at a corresponding predefinable simulation time point.

[0021] In the following, it is specified that within a simulation time window (rollout frame), all calculations are performed according to the sequence of the control pattern. This means that the simulation time window includes all simulation times required to perform the calculations according to the sequence of the control pattern. If all layers determine their intermediate variables independently of the sequence, the simulation time window includes only one simulation time at which all layers determine their intermediate variables. In particular, these layers determine their intermediate variables at this simulation time only if their input variables are supplied to them.

[0022] Preferably, the delay (d) of a connection, in particular a neuron, depends on the number of simulation time windows that are implemented starting from the simulation time window in which the intermediate variables of the first layer are determined until the simulation time window in which the intermediate variables of the second layer connected to the first layer via the connection are determined. The delay (d) can correspond to the number of simulation time windows of the deep neural network that is expanded over time according to the control mode, which are implemented until the intermediate variables of the subsequent layers connected via the connection have been determined.

[0023] In the following, it is specified that the difference between two immediately consecutive simulation time points corresponds to a time step. Preferably, each time step in the deep neural network corresponds to a physical time interval Δt of the spiking neural network. During this physical time interval Δt, the spiking neural network is given a single input variable (e.g., a single image of a video).

[0024] If all layers determine their intermediate variables independently of this order, the simulation time window contains only one simulation time point and can therefore have a duration of one time step, preferably the duration of the physical time interval Δt. Thus, a connection between two layers separated by d time steps (in a deep neural network developed according to this control scheme, all layers are independent) results in a delay dΔt in the spiking neural network.

[0025] The predefinable time window for the integration over time can include a plurality of time steps.

[0026] It should be noted that in spiking neural networks, more than one delay can also be assigned to these connections or neurons. This method can also be applied to temporal convolutional networks, where multiple delays can be used for the temporal filter. If a connection spans, for example, three simulation time windows, then corresponding delays d of d=0, d=1, and d=2 must be selected for the corresponding regions of the filter.

[0027] It is further proposed that the deep neural network has at least one bridging connection. A delay is assigned to the bridging connection as a function of the control mode and / or as a function of the number of layers of the spiking neural network that are bridged by the bridging connection.

[0028] The advantage of jumper connections in spiking neural networks is that they significantly improve the temporal integration of the spiking neural network. When converting deep neural networks with jumper connections into equivalent spiking neural networks, the problem of information being processed in the deep neural network at times different from those specified in the spiking neural network often arises. Introducing a delay (dΔt) ensures that information from different neurons along the connections and the jumper connections reaches the correct layer / neuron in the spiking neural network at the correct time. This control mode allows this to be taken into account during the teaching process. This approach also allows for flexible use of the control mode, for example, depending on the computing resources available during the teaching process, so that the temporal integration can still be fully taken into account during the teaching process.

[0029] Additional cross-connections can be added to deep neural networks. This has the advantage that the integration over time becomes much more precise.

[0030] A skip connection can be a forward or backward skip connection or a recurrent connection connecting the input and output of the same layer. The advantage of a skip connection is that it increases the receptive field of the spiking neural network over time.

[0031] It is also proposed to use a receptive field that is spread out over space and time (spatio-temporal receptive field). It is proposed that this receptive field that is spread out over space and time be extended by at least one simulation time window to enable integration over time. This receptive field that is spread out over space and time can be specified by the control mode. Considering the corresponding application of spiking neural networks, the receptive field over time should be selected such that processes over time can be resolved.

[0032] It is also proposed to quantize the parameters and / or intermediate quantities of a deep neural network during the teaching process. In addition to enabling faster and more efficient teaching, this quantization has the surprising advantage that it has a positive impact on the conversion of deep neural networks into spiking neural networks. This is because in order to resolve small differences in activations, the number of simulation steps per simulation time window must be set to a high value, which in turn leads to a higher firing rate and lower energy efficiency. In order to reduce the required simulation steps per time step, this inherent limitation of low resolution at low firing rates is accommodated by quantizing the activations during the teaching process. Furthermore, simulations have shown that by quantizing the deep neural network, the firing rate converges more quickly to the target firing rate of the spiking neural network, because the quantization emphasizes a number of specific activations.

[0033] Quantization can be understood as using a predeterminable number of bits to represent a parameter. These parameters (such as weights or thresholds) are preferably quantized using less than 32 bits, or 16 bits, 8 bits, or 4 bits. Preferably, linear quantization is used during teaching. During teaching, the quantization resolution can depend on the maximum activation ( The maximum activation ( ) can be an exponential moving average of the standard deviation of the positive activations during the forward propagation of the teaching. It is conceivable that: each layer / neuron ( ) has its own maximum activation ( ).

[0034] It is also proposed that this control mode corresponds to a streaming control mode (English: Streaming Rollout). This streaming control mode is understood to mean that all layers / neurons of the deep neural network are run independently of the order and that these layers / neurons determine their intermediate variables based on the input variables at each simulation time point. This streaming control mode is advantageous because the teaching becomes more efficient in terms of computing and storage. This is because the layers of the deep neural network are always active due to the parallel operation, which leads to a higher execution speed and a higher response speed of the deep neural network. This way of running a deep neural network essentially corresponds to the way of running a spiking neural network, because the firing rate then essentially corresponds to the activation of the deep neural network. Therefore, this approach makes it possible to generate a spiking neural network with minimal effort, and also because the delay can then be uniformly set to 1Δt.

[0035] It is also proposed to use spatial dropout during training. In the case of spatial dropout, entire filters are temporarily deactivated. For example, an image may have dimensions (3, 30, 40) (color, y, x) at the input and be processed using a convolutional layer with 6 kernels / channels. The output dimensions are then (6, 30, 40). In the case of spatial dropout, at least one of these 6 kernels is randomly deactivated, preventing information from being transmitted through that path. Neural networks learn by processing independent information through different channels. This property is preserved when converted to spiking networks. The advantage in this case is that this approach results in fewer neurons firing, thereby improving energy efficiency.

[0036] It is further proposed that after this conversion, a spiking neural network is run, in particular as a function of the delay, and that the input variable of the spiking neural network is a sequence or time series of event-based photographs, in particular event-based photographs of an event-based camera. Alternatively, the input variable may be a video sequence. Sensor values ​​detected by sensors in other ways are also conceivable.

[0037] The combination of a spiking neural network, characterized by its particularly energy-efficient reasoning and fast processing, with an event-based camera, which can also capture images particularly energy-efficiently and quickly, results in a particularly fast and energy-efficient system. This system can be used anywhere, preferably in energy-scarce situations and / or in situations where rapid decisions or classifications must be made. For example, a fast implementation of the network is advantageous in terms of hazard recognition and / or in terms of localizing fast objects. The spiking neural network preferably has two channels, one containing all "on" events of the event-based camera and one containing all "off" events of the event-based camera, so that a higher amount of information is available at the input of the spiking neural network.

[0038] It is further proposed that during the operation of the spiking neural network, the weights of the connections of the spiking neural network are scaled over time, in particular so that a short pulse arriving early within a time step Δt leads more quickly to the emitted short pulse. Preferably, the values ​​of the weights are reduced over this time step Δt. This allows the spiking neural network to determine its output variable particularly quickly, energy-efficiently, and reliably.

[0039] It is further proposed that control variables for controlling actuators of a technical system be determined or provided based on the determined output variables of the spiking neural network. The technical system may be, for example, an at least partially autonomous machine, an at least partially autonomous vehicle, a robot, a tool, a factory machine, or an aerial vehicle such as a drone.

[0040] In another aspect, a computer program is provided. The computer program is configured to implement one of the methods described above. The computer program includes instructions that, when executed on a computer, cause the computer to implement all the steps of one of the methods described above. Also provided is a machine-readable storage module on which the computer program is stored. Also provided is a device configured to implement one of the methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Embodiments of the above aspects are shown in the accompanying drawings and are further explained in the following description.

[0042] Figure 1 A schematic diagram showing a flow chart of a method for creating a spiking neural network;

[0043] Figure 2 A schematic diagram showing an embodiment of an apparatus for creating a spiking neural network is shown. DETAILED DESCRIPTION

[0044] Figure 1 Shows the method used to create a spiking neural network (EnglishS piking N eural N Schematic diagram of the method (10) of etwork, SNN).

[0045] The method begins at step 11. In this step, a deep neural network is provided. The deep neural network includes multiple layers, each of which is connected to another. Each of these layers may include multiple neurons. The deep neural network may be a previously taught deep neural network or a deep neural network whose parameters are randomly initialized, for example.

[0046] In the next step 12, a control mode is assigned to the deep neural network. This control mode specifies the order in which the layers determine their intermediate variables. For example, the control mode can specify that the layers calculate their output variables sequentially. In this case, each layer must wait until it is supplied with the same input variables so that it can then determine its intermediate variables. For example, the control mode can also specify that the layers are executed completely in parallel (see Streaming Rollout).

[0047] After step 12 is completed and if the provided deep neural network has not been taught after step 11, step 13 follows. If the neural network has already been taught, this step is skipped. In step 13, the deep neural network is taught using the control mode. In this case, using training data comprising training input variables and corresponding assigned training output variables, the deep neural network is taught so that it determines the corresponding assigned training output variables of these training input variables based on the training input variables. In this case, the parameters of the deep neural network can be adapted with the help of a gradient descent method so that the deep neural network determines the corresponding assigned training output variables. The gradient descent method can optimize the "categorical cross entropy" cost function based on the parameters of the deep neural network. Preferably, during the teaching, input variables, such as images, are applied to the deep neural network multiple times in sequence, and the deep neural network determines the output variables for each of the input variables multiple times based on the control mode. Alternatively, a sequence of input variables can also be used.

[0048] After step 13 or step 12 is completed, step 14 follows. In this step, the taught deep neural network is converted into a spiking neural network. During this conversion, the architecture and parameterization of the deep neural network are used to create a spiking deep neural network. The activations of the neurons of the deep neural network can be converted into proportional firing rates of the neurons of the spiking neural network. For a detailed explanation of the method of this conversion, refer to the document "Conversion of Continuous-Valued Deep Networks to Efficient Event-DrivenNetworks for Image Classification" cited at the beginning. In addition, delays are respectively assigned to the connections of the spiking neural network according to the control mode used of the deep neural network. Preferably, an argmax output layer of the spiking neural network is used, which argmax output layer is generated at a predefined time interval t readout All incoming spikes are counted during the period and a mathematical operator is applied to the counted spikes of the neurons of the argmax output layer. .

[0049] An optional step 15 can then be carried out, in which the spiking neural network is run according to the assigned delays.

[0050] Spiking neural networks can be used, for example, in an at least partially autonomous robot. The at least partially autonomous robot can be, for example, an at least partially autonomous vehicle. In another embodiment, the at least partially autonomous robot can be a service robot, an assembly robot, or a stationary production robot, or alternatively, an autonomous aerial vehicle, such as a drone.

[0051] In a preferred embodiment, the at least partially autonomous vehicle comprises an event-based camera. The camera is connected to a spiking neural network, which determines at least one output variable based on the provided camera image. The output variable can be forwarded to a control unit. The control unit controls an actuator based on the output variable, preferably controlling the actuator so that the vehicle (10) performs a collision-free maneuver. In a first embodiment, the actuator can be a motor or a braking system of the vehicle. In another embodiment, the partially autonomous robot can be a tool, a factory machine or a manufacturing robot. The material of the workpiece can be classified with the aid of the spiking neural network. In this case, the actuator can be, for example, a motor that drives a grinding head.

[0052] Figure 2A schematic diagram of a device (20) for teaching a deep neural network, in particular for carrying out a teaching step, is shown. The device (20) comprises a training module (21) and a module to be trained (22). The module to be trained (22) contains a deep neural network. The device (20) teaches the deep neural network based on output variables of the deep neural network and preferably using predefined training data. Advantageously, the training data include a plurality of respectively labeled detected image or sound sequences, text fragments, event-based signals, radar signals, lidar signals, or ultrasound signals. During the teaching process, the parameters of the deep neural network stored in a memory (23) are adapted.

[0053] The device further comprises a computing unit (24) and a machine-readable storage element (25). A computer program can be stored on the storage element (25), the computer program comprising instructions which, when executed on the computing unit (24), cause the computing unit (24) to execute a method for creating a spiking neural network, such as Figure 1 As shown in .

Claims

1. A method for creating a spiking neural network by converting a deep neural network into a spiking neural network, the method comprising the steps of: - assigning a predetermined control pattern to the deep neural network, The control mode represents the order of calculation, and the layers or neurons of the deep neural network determine the intermediate parameters of the layers or neurons in the order, and wherein the control mode characterizes which layers or neurons of the deep neural network determine their intermediate parameters independently of the order; - teaching the deep neural network using the control mode; - converting the deep neural network into the spiking neural network, wherein the delays assigned to the connections and / or neurons of the spiking neural network are selected according to the control mode, in, The delays of the connections of the spiking neural network are adapted according to the control mode so that temporal adjustment of the information can be achieved when the information is transmitted through the spiking neural network, wherein it is ensured that the information is present at the correct corresponding neuron of the spiking neural network at the correct point in time, wherein, when the spiking neural network is run, the input parameter of the spiking neural network is a sequence of event-based photographs, or the input parameter is a video sequence.

2. The method of claim 1, wherein the deep neural network has at least one crossover connection, When converting the deep neural network into the spiking neural network, the assigned delays of the jumper connections of the spiking neural network are selected as a function of the control mode and / or as a function of the number of layers of the deep neural network that are jumpered by the jumper connections.

3. The method according to claim 1 or 2, wherein parameters and / or intermediate variables of the deep neural network are quantized during the teaching. The method according to claim 1 , wherein the control mode corresponds to a flow control mode. The method according to claim 1 , wherein spatial signal loss is used in performing the teaching.

6. The method according to claim 1 or 2, wherein the training input parameters of the deep neural network are respectively arranged multiple times in succession as a sequence of identical input parameters, or the training input parameters are a sequence of temporally continuous input parameters, and The deep neural network is taught based on the sequence.

7. The method according to claim 1 or 2, wherein after the conversion, the spiking neural network is run, During the operation of the spiking neural network, the weights or connections of the spiking neural network are scaled within a time step during which an input variable is applied to the input of the spiking neural network.

8. The method of claim 7, wherein after the converting, the spiking neural network is run according to the delay.

9. The method according to claim 1 or 2, wherein an input variable of the spiking neural network when running the spiking neural network is a sequence of event-based pictures of an event-based camera. 10 . A computer program product, comprising a computer program, the computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claim 1 . 11 . A machine-readable storage element having a computer program stored thereon, the computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claim 1 . 12 . A device configured to carry out the method according to claim 1 .

13. A spiking neural network obtained by implementing the method according to any one of claims 1 to 9.

Images