Spiking Neural Network Device

Abstract

Reduce the attenuation of backpropagation errors through activation functions. [Solution] A spiking neural network device 1 that learns by backpropagating error information from neurons in a later layer to neurons in an earlier layer includes a calculation unit 11 that calculates the differential value of the activation function used when generating spikes and backpropagating error information from neurons in a later layer to neurons in an earlier layer, a teacher signal generation unit 12 that generates a teacher signal for neurons in the earlier layer based on the differential value calculated by the calculation unit 11, and a learning rate adjustment unit 13 that attenuates, at a predetermined period, the learning rate that determines how much to update learning parameters optimized by learning, and the activation function is a linear function with a slope that is a power of 2.

Description

[Technical Field]

[0001] The present invention relates to a spiking neural network device that reduces the attenuation of backpropagation of errors due to activation functions. [Background technology]

[0002] Machine learning is generally considered a type of artificial intelligence (AI), a computer algorithm that automatically improves by learning from experience. By training using data called "training data," the algorithm can automatically learn parameters that are difficult to specify in a program, and use the learning results to perform a desired task.

[0003] A neural network is one type of AI architecture. It is an algorithm created with the aim of imitating the structure and function of a neural network, and is made up of mathematically modeled nerve cells (neurons) connected to a network structure.

[0004] One commonly known type of neural network is the spiking neural network (SNN), which is a neural circuit model that models the firing (spikes) of nerve cells (neurons) as a time series and transmits and processes information as a spike train.

[0005] Patent Document 1 discloses a technology related to a spiking neural network (SNN) that includes a first phase for propagating data from an input layer to an output layer and a second phase for backpropagating an error from the output layer to the input layer, and the spiking artificial neuron includes a first input / output interface capable of receiving a binary or ternary error signal weighted by a synaptic coefficient to execute the second backpropagation phase, an error calculation module configured to calculate a binary or ternary local error signal from an estimated value of a differential coefficient of an equivalent activation function implemented by the neuron during execution of the first data propagation phase, and a second input / output interface capable of propagating the binary or ternary local error signal to a plurality of synapses in the form of spikes. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 7433307 [Non-patent literature]

[0007] [Non-Patent Document 1] Minoru Motoki, Hirohito Shintani, Kazunori Matsuo, Thomas Martin McGinnity, “Utilization of SAM-based Network for Developing Function Approximation”, [online], December 2022,<URL: https: / / www.dline.info / fpaper / jdim / v20i4 / jdimv20i4_3.pdf> Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the technique described in Patent Document 1, a binary or ternary local error signal is calculated from an estimated value of the differential coefficient of the equivalent activation function, which causes a problem of attenuating the propagation of the error signal.

[0009] Here, the step function g(u) shown in Figure 1(a) is often used as the activation function. The differential value of the step function g(u) used in the first data propagation phase is then used in the second backpropagation phase.

[0010] Therefore, the ideal derivatives of a step function (activation function) are α = positive infinity, β = 0, and γ = 0, but in reality, values ​​such as α = 1, β = 1 / 16, and γ = 1 / 16 are used, as shown in Figure 1(b). As such, the more ideal the derivative of a step function (activation function), the smaller the value by which it is multiplied, so the derivative of the step function attenuates the propagation of errors to the previous layer.

[0011] This step function is used to calculate the error of the hidden layer in a SAM-type spiking neural network (SNN), for example, as shown in Equation (23) of Non-Patent Document 1.

[0012] As shown in equation (22) of Non-Patent Document 1, when the value of equation (23) becomes small, the teacher signal T j (2) (t) is the output X j <2> Since it tends to have the same polarity as (t), the teacher signal T j (2) (t) makes it harder to request corrections from the previous layer, which results in a decrease in the amount of backpropagation of errors to the previous stage. This causes the gradient to disappear, making it harder to learn the hidden layer parameters.

[0013] The present invention has been made in view of the above-mentioned problems, and has as its object to provide a spiking neural network device that reduces the attenuation of backpropagation of errors due to activation functions. [Means for solving the problem]

[0014] In order to achieve the above object, a first feature of a spiking neural network device according to the present invention is: A spiking neural network device that learns by backpropagating error information from neurons in a later layer to neurons in a previous layer, a differential calculation unit that calculates a differential value of an activation function used when generating spikes and backpropagating error information from neurons in a subsequent layer to neurons in a previous layer; a teacher signal generating unit that generates a teacher signal to the neurons in the previous layer based on the differential value calculated by the differential calculating unit; Equipped with The activation function is a linear function with a power of two as its slope. A second feature of the spiking neural network device according to the present invention is that: The preceding layer is a hidden layer. A third feature of the spiking neural network device according to the present invention is that The learning parameter adjusting unit further comprises a learning rate adjusting unit that attenuates, at predetermined intervals, a learning rate that determines how much the learning parameter optimized by the learning is updated. A fourth feature of the spiking neural network device according to the present invention is that The learning parameter is at least one of a connection weight of the neuron and a threshold value for firing the spike. A fifth feature of the spiking neural network device according to the present invention is The predetermined period is set in units of one or more epochs. A sixth feature of the spiking neural network device according to the present invention is that The learning rate is set to be halved every predetermined period. [Effects of the Invention]

[0015] The spiking neural network device according to the present invention can reduce the attenuation of backpropagation of errors due to activation functions. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is an explanatory diagram illustrating the calculation of a derivative value g'() according to the prior art. (a) shows the activation function g() used in the prior art, and (b) shows the derivative value g'() of the activation function g() shown in (a). [Figure 2] FIG. 1 is a conceptual diagram of a spiking neural network device according to a first embodiment of the present invention. [Figure 3] 1A and 1B are explanatory diagrams illustrating the calculation of a differential value g'() by a calculation unit 11 included in a spiking neural network device according to a first embodiment of the present invention. (a) shows an activation function g() used for convenience during backpropagation in the spiking neural network device 1, and (b) shows a differential value g'() of the activation function g() shown in (a). [Figure 4] FIG. 2 is an explanatory diagram illustrating the processing by the learning rate adjuster 13 included in the spiking neural network device according to the first embodiment of the present invention. [Figure 5] 1 is a flowchart showing the processing content of a spiking neural network device according to a first embodiment of the present invention. [Figure 6] This shows the accuracy rate for unknown data (text data) when a handwritten digit dataset (scikit-learn digits) is trained on a 3-layer and 5-layer spiking neural network device 1, with the training data:test data divided into 8:2. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same or equivalent parts and components are designated by the same or equivalent reference numerals throughout the drawings. However, it should be noted that the drawings are schematic and may differ from the actual product. Furthermore, the drawings may include parts with different dimensional relationships and ratios.

[0018] Furthermore, the embodiments shown below are merely examples of devices that embody the technical concept of the present invention, and the technical concept of the present invention does not limit the arrangement of each component to that shown below. Various modifications can be made to the technical concept of the present invention within the scope of the claims.

[0019] Example 1 FIG. 2 is a conceptual diagram of a spiking neural network device according to a first embodiment of the present invention. As shown in FIG. 2, the spiking neural network device 1 comprises multiple layers C of interconnected spiking neurons. e , C l , C s The network contains at least one input layer C e , output layer C s and at least one hidden layer (intermediate layer) C l The input layer C e Neuron N i,e Each receives input data 101 at its input terminal. l However, the number of hidden layers is not limited to one, and it is of course possible for the device to have multiple hidden layers.

[0020] Each neuron in each layer is connected to all neurons in the previous or next layer through its input and / or output terminals. i,e and N i,l The connections between them are made by connections 102, 103, and 104, each made by a synapse.

[0021] The spiking neural network device 1 has two separate phases: an input layer C e to output layer C s The first phase propagates data to the output layer C s to input layer C e It includes a second phase that backpropagates the error, and the synaptic weights are updated layer by layer.

[0022] In the first data propagation phase, training data, e.g., an image or a sequence of images, is fed to the input layer C e The data is provided as input to neurons in the network and propagated through the network. In the case of spiking neural networks, the data is encoded in the form of asynchronous spikes. The spikes correspond to binary or ternary signals. In other words, they can be signed or unsigned spikes.

[0023] During the first phase, each neuron in the previous layer (or input layer C e Each neuron implements a function that integrates the spikes it receives from neurons in the layer (spikes encoded from the input data to the neurons in the layer). The integration function essentially accumulates spikes weighted by artificial synaptic weights. Each neuron also implements an activation function that generates and propagates spikes to neurons in the next layer based on a comparison of the integration variable with one or two activation thresholds.

[0024] Depending on the neuron model chosen, the integral and activation functions may differ. In particular, a leaky flow may be implemented by the neuron to decay the integral variable over time if the neuron does not receive any spikes. s Neuron N i,s performs additional processing, resulting in neuron N i,s The integration result of the spike received by the output layer C s , which corresponds to the final state of the neuron and calculates the error between it and the expected or target value that one wishes to obtain in relation to the training input data.

[0025] In the second backpropagation phase, the output layer C s The neurons in the hidden layer C l The neuron calculates the local error (error information) from the error backpropagated from the previous layer, and then sends the local error (error information) to the previous layer, the input layer C e In parallel, each neuron calculates the updated weight of the synapse connected to it from the local error (error information) and updates the synapse. This process is performed in the input layer C e This continues through each layer of neurons until the penultimate layer, which is responsible for updating the weights of the synapses connecting to the

[0026] The spiking neural network device 1 also has a control unit 10. The control unit 10 is an arithmetic module configured by hardware such as a processor such as a CPU (Central Processing Unit), a memory, and other electronic circuits, or software such as a program having the functions thereof, or a combination of these. The control unit 10 functionally comprises a calculation unit 11, a teacher signal generation unit 12, and a learning rate adjustment unit 13. The calculation unit 11 calculates the internal potential and output of each neuron in each layer.

[0027] Hidden layer C l (Second layer) At time t, the jth neuron receives the ith input value X i <1> When a spike at (t) is received, the connection weight W ji <2> and integrate them by the number of inputs n1, and calculate the internal potential V j <2> The internal potential U is the potential obtained by multiplying (t-1) by the attenuation coefficient a. j <2> If (t), it can be expressed by the following (Equation 1). Then, as shown in (Equation 2), the calculation unit 11 calculates the internal potential U j <2> (t) is passed through the activation function g() to the output Xj <2> Calculate (t).

[0028] Furthermore, if we assume that the internal potential decreases by p due to firing, the internal potential after firing V j <2> (t) can be expressed using (Equation 3), where the superscript <> indicates the layer number. However, V j <2> Let (0)=0.

[0029]

number

[0030]

number

[0031]

number

[0032] Similarly, the output layer C s For the (third layer), the calculation unit 11 calculates the internal potential U k <3> (t) is calculated, and the output X is calculated using (Equation 5). k <3> (t) is calculated, and the internal potential after firing, V, is calculated using (Equation 6). k <3> (t) is calculated. However, V k <3> Let (0)=0.

[0033]

number

[0034]

number

[0035]

number

[0036] The threshold value θ for spike firing in a spiking neural network is set for each neuron, and is expressed as θ j <2> ,θ j <3> It is variable as a learning parameter. Furthermore, the activation u j <2> (t) is defined as follows:

[0037]

number

[0038]

number

[0039]

number

[0040] However, T j <2> (t) is the hidden layer C l The calculation unit 11 calculates g'() using the following (Formula 10) and (Formula 11): where g'() is the differential value of the activation function g() used when generating spikes and backpropagating local errors (error information) from neurons in the later layer to neurons in the earlier layer.

[0041]

number

[0042]

number

[0043] Here, as mentioned above, if the differential value of the step function is used for g'(), the amount of backpropagation of error to the previous stage will be attenuated, making it difficult to learn the parameters of the hidden layer.

[0044] Therefore, in order to calculate the output, the calculation unit 11 calculates the differential value g'() of the activation function g() used when generating spikes and backpropagating local errors (error information) from neurons in the later layer to neurons in the earlier layer.

[0045] FIG. 3 is an explanatory diagram illustrating the calculation of the differential value g'() by the calculation unit 11 included in the spiking neural network device 1 according to the first embodiment of the present invention. FIG. 3(a) shows the activation function g() used for convenience during backpropagation in the spiking neural network device 1, and FIG. 3(b) shows the differential value g'() of the activation function g() shown in FIG. 3(a). The X-axis in FIGS. 3(a) and 3(b) represents the activation level u. The activation function g() shown in FIG. 3(a) is the activation function that is used to calculate the differential value g'() shown in FIG. 3(b), and a step function is used during forward propagation.

[0046] As shown in FIG. 3(a), the activation function g() used for convenience in the backpropagation of the spiking neural network device 1 is not a step function but a linear function whose slope is a power of 2. The slope of the activation function L13 shown in FIG. 3(a) is 2 1 = 2, and the slope of the activation function L12 shown in Figure 3(a) is 2 0 = 1, and the slope of the activation function L13 shown in Figure 3(a) is 2 -1 =0.5. Preferably, the slope of the activation function g() is equal to or greater than 1. Note that although the intercept is set to 0 here, it may be either a positive or negative value.

[0047] When the activation function g() shown in FIG. 3(a) is differentiated, the differential value g'() becomes the differential values ​​L23, L22, and L21 as shown in FIG. 3(b). The differential value L23 obtained by differentiating the activation function L13 shown in FIG. 3(a) is 2 1 (=2), and the differential value L22 obtained by differentiating the activation function L12 shown in Figure 3(a) is L22=2 0 (=1), and the differential value L21 obtained by differentiating the activation function L11 shown in Figure 3(a) is L21=2- 1 (=0.5).

[0048] Then, the teacher signal generating unit 12 generates the hidden layer C l (Second layer) neurons. l (2nd layer) neurons, but also input layer C e A teaching signal to the neuron may be generated.

[0049] In this way, by using a linear function with a power of 2 as the gradient for the activation function g() used for convenience during backpropagation, it is possible to prevent the value shown in (Equation 11) from becoming extremely small. As a result, when the value shown in Equation (23) becomes small, the teacher signal T j (2) (t) is the output X j <2> Since the polarity is likely to be opposite to that of (t), the teacher signal T j (2) (t) makes it easier to request corrections from the previous layer, and can properly backpropagate errors to the previous stage. This allows for proper learning of hidden layer parameters.

[0050] Furthermore, by making the activation function g() a linear function with a slope that is a power of 2, a multiplier is not required, even when implementing it in a digital circuit such as an FPGA, and the circuit size can be kept small.

[0051] The learning rate adjuster 13 periodically attenuates the learning rate, which determines the degree to which the learning parameters optimized by learning are updated. Here, the learning parameter is at least one of the neuron weight W and the spike firing threshold θ.

[0052] For example, the connection weight W from the mth neuron in the (l-1)th layer to the nth neuron in the lth layer n <l>< / l> m is updated using the following (Equation 12): η is the learning rate of the connection weight W. Also, the threshold θ n <l>< / l> is updated using (Equation 13), where ι is the learning rate of the threshold θ.

[0053]

number

number

[0054] FIG. 4 is an explanatory diagram illustrating the processing by the learning rate adjuster 13 included in the spiking neural network device 1 according to the first embodiment of the present invention.

[0055] 4, the learning rate adjuster 13 attenuates at least one of the learning rate η of the connection weight W and the learning rate ι of the threshold value θ every 2 epochs. Here, an epoch refers to a unit in which the entire data set used for learning is used in one learning session.

[0056] Specifically, the learning rate adjuster 13 sets the learning rate (η or ι) to "0.25" (symbol R01) between epochs E01 and E02, that is, from time t1 to time t2.

[0057] Thereafter, the learning rate adjuster 13 halves the learning rate (η or ι) from "0.25" (R01) to "0.125" (R02) between epochs E03 and E04, i.e., from time t2 to time t3. Furthermore, the learning rate adjuster 13 halves the learning rate (η or ι) from "0.125" (R02) to "0.0625" (R03) between epochs E05 and E06, i.e., from time t3 to time t4, and halves the learning rate (η or ι) from "0.0625" (R03) to "0.03125" (R04) between epochs E06 and E07, i.e., from time t4 to time t5.

[0058] In this way, the learning rate adjuster 13 sets the learning rate (η or ι) to be halved every one or more epochs. This allows the learning parameters to change significantly in the early stages of learning by setting the learning rate relatively high. This allows for a broad search in the early stages, facilitating convergence to a solution close to the optimal solution. Meanwhile, in the middle to later stages of learning, the learning rate is gradually decreased to suppress changes in the learning parameters, stabilizing learning and improving convergence accuracy.

[0059] FIG. 5 is a flowchart showing the processing contents of the spiking neural network device 1 according to the first embodiment of the present invention. In step S101, the calculation unit 11 calculates the loss of the next layer, i.e., the output X k <3> (t) to T k Calculate the value obtained by subtracting (t). In step S103, the calculation unit 11 calculates the differential value g'(uk(t)) of the activation function g(uk(t)) of the next layer. In step S105, the calculation unit 11 calculates the transposed matrix of the parameters of the next layer, i.e., the connection weights W ji <3> Calculate (t). In step S107, the calculation unit 11 calculates the gradient of the learning parameters by multiplying the values ​​calculated in steps S101 to S103 and accumulating only the input n3 using the above-mentioned (Equation 11). In step S109, the calculation unit 11 updates the learning parameters using the above-mentioned (Equation 12) and (Equation 13).

[0060] FIG. 6 is a diagram showing the effect of the spiking neural network device 1 according to the first embodiment of the present invention. Figure 6 shows the accuracy rate for unknown data (text data) when a handwritten digit dataset (scikit-learn digits) was trained on a 3-layer and 5-layer spiking neural network device 1, with the training data divided into 8:2 training data and test data.

[0061] When character recognition was performed using a three-layer spiking neural network device, which is conventional technology for comparison, the recognition rate was 89.4%, whereas when character recognition was performed using the spiking neural network device 1 of Example 1 of the present invention, the recognition rate was 91.1%, demonstrating an improvement.

[0062] Furthermore, when character recognition was performed using a five-layer spiking neural network device, which is a conventional technology for comparison, the recognition rate was 56.4%, whereas when character recognition was performed using the spiking neural network device 1 of Example 1 of the present invention, the recognition rate was 93.1%, a significant improvement.

[0063] In the spiking neural network device 1 according to the first embodiment of the present invention, the hidden layer parameters are appropriately learned by appropriately backpropagating errors to the previous stage, and therefore the greater the number of layers, the greater the effect and the greater the susceptibility to this effect. [Explanation of symbols]

[0064] 1. Spiking Neural Network Device 10 Control Unit 11 Calculation unit (differential calculation unit) 12 Teacher signal generation section 13 Learning rate adjustment unit

Claims

1. A spiking neural network device that learns by backpropagating error information from neurons in a later layer to neurons in a previous layer, a differential calculation unit that calculates a differential value of an activation function used when generating spikes and backpropagating error information from neurons in a subsequent layer to neurons in a previous layer; a teacher signal generating unit that generates a teacher signal to the neurons in the previous layer based on the differential value calculated by the differential calculating unit; Equipped with The activation function is a linear function with a slope that is a power of 2. A spiking neural network device characterized by:

2. 2. The spiking neural network device according to claim 1, wherein the previous layer is a hidden layer.

3. a learning rate adjusting unit that attenuates a learning rate that determines how much the learning parameters optimized by the learning are updated at predetermined intervals; 2. The spiking neural network device according to claim 1, further comprising:

4. The learning parameter is at least one of the connection weight of the neuron and the threshold for firing the spike.

4. The spiking neural network device according to claim 3.

5. The predetermined period is set in units of one or more epochs.

4. The spiking neural network device according to claim 3.

6. The learning rate is set to be halved every predetermined period.

4. The spiking neural network device according to claim 3.

Images