Method and System for Spiking Neural Network Based Conversion Aware Training

Abstract

The present invention relates to a method and system for learning a spiking neural network based on transformation recognition learning. According to the present invention, the method comprises an ANN generation step of generating an analog artificial neural network (ANN) model and inputting variable data, a transformation recognition learning step of simulating a spiking neural network (SNN) model using one or more activation functions in the ANN model, and an SNN generation step of generating an SNN model by correcting the parameters and weights of a layer based on the simulation results.

Description

Technology Field

[0001] The present invention relates to a method and system for training a Spiking Neural Network based on a transformation recognition learning model. Background Technology

[0002] Neuromorphic technology is a technology designed to hardware-mimicking the human nervous system and was proposed to overcome the limitations of existing computing architectures, which are much less efficient and consume more power than humans in performing cognitive processing functions. Consequently, neuromorphic technology is attracting attention for operating edge devices, which have limited power and battery life, with low power and low energy consumption.

[0003] A representative example of neuromorphic technology is the Spiking Neural Network (SNN). An SNN is a neural network designed to mimic the characteristics of the human brain, which consists of a neuron-synapse structure where synapses connecting neurons transmit information via electrical signals in the form of spikes. Such SNNs process information based on the time difference in which spike signals are transmitted. Specifically, SNNs transmit information using binary spike signals, conveying information in a form where a set of binary spikes (either 0 or 1) is composed of a single unit. These signals are transmitted to neurons through the neurons within the SNN, and whether a spike occurs is determined by differential equations representing various biological processes. Specifically, when a spike arrives at the input of a neuron, the input spike is decoded and processed with synaptic weights, and the result is accumulated in the neuron's membrane potential. If the accumulated membrane potential value exceeds a threshold, the neuron generates an output spike, which is then transmitted to the next neuron. During this process, the membrane potential of the neuron that generated the spike is initialized to 0.

[0004] As such, since the SNN operation operates only when a spike occurs, it enables low-power hardware implementation. Prior art literature

[0006] Korean Registered Patent No. 10-2344678 The problem to be solved

[0007] The present invention aims to solve the aforementioned problems, and the objective of the present invention is to provide a spiking neural network training method and system that minimizes data loss occurring during the conversion of ANN training data into SNN training data.

[0008] In addition, the present invention provides a spiking neural network learning method and system that applies an ANN learning model similar to an SNN learning model. means of solving the problem

[0009] In one embodiment of the present invention, a method for learning a spiking neural network based on transformation recognition learning is provided, comprising: an ANN generation step of generating an analog artificial neural network (hereinafter ANN) model and inputting variable data; a transformation recognition learning step of simulating a spiking neural network (hereinafter SNN) model using one or more activation functions in the ANN model; and an SNN generation step of generating an SNN model by correcting the parameters and weights of a layer based on the simulation results.

[0010] In addition, to achieve the above-mentioned purpose, another embodiment of the present invention provides a spiking neural network learning system based on transformation recognition learning, comprising: an ANN generation unit that generates an analog artificial neural network (hereinafter ANN) model and inputs variable data; a transformation recognition learning unit that simulates a spiking neural network (hereinafter SNN) model using one or more activation functions in the ANN model; and an SNN generation unit that generates an SNN model by correcting the parameters and weights of a layer based on the simulation results. Effects of the invention

[0011] According to the present invention, data loss occurring during the conversion of ANN training data into SNN training data can be minimized.

[0012] In addition, it is possible to perform high-performance training using SNN training models that can only be processed by existing ANN training models, while enabling hardware operation at low power. Brief explanation of the drawing

[0013] Figure 1 is a diagram illustrating a conventional ANN-to-SNN conversion technique. FIG. 2 is a flowchart of a method for learning a spiking neural network based on transformation recognition learning according to an embodiment of the present invention. FIG. 3 is a flowchart of a transformation recognition learning step according to one embodiment of the present invention. FIG. 4 is a diagram briefly illustrating a spiking neural network learning method based on transformation recognition learning according to one embodiment of the present invention. Figures 5(a) to 5(c) are graphs of simulation results for a spiking neural network learning method based on transformation recognition learning according to an embodiment of the present invention. FIG. 6 is a configuration diagram of a spiking neural network learning system based on transformation recognition learning according to one embodiment of the present invention. FIG. 7 is a block diagram of a processing device applying a transformation recognition learning-based spiking neural network learning system according to one embodiment of the present invention. Specific details for implementing the invention

[0014] In order to explain the technical concept of the present invention in detail so that a person skilled in the art can easily implement it, embodiments of the present invention will be described below with reference to the attached drawings.

[0015] Figure 1 illustrates a general ANN-to-SNN conversion technique.

[0016] As shown in Figure 1, to implement a high-performance SNN, a general analog Artificial Neural Network (ANN) is pre-trained by applying backpropagation learning, and then converted into an SNN. This technique is performed because appropriate processing is required due to the difference in data representation between SNNs and ANNs, and it is a technique devised because it is difficult to apply backpropagation learning, which is the learning method of high-performance ANNs, to SNNs. In other words, it is a conversion technique in which an ANN is implemented before running the SNN to obtain updated weights through backpropagation learning, and said weights are applied to the SNN.

[0017] While conventional ANN-to-SNN conversion techniques demonstrate high accuracy compared to data trained solely on SNNs, they suffer from data loss when converting the analog values ​​of ANN data into discrete spikes occurring at specific points in time. To minimize this data loss, methods are employed where ANNs and SNNs are trained separately, their outputs are compared, and weights are normalized or neural network layer parameters are adjusted based on the comparison results. However, this approach places a burden on hardware operation, highlighting the need for an energy-efficient ANN-to-SNN conversion technique.

[0018] FIG. 2 is a flowchart of a method for learning a spiking neural network based on transformation recognition learning according to an embodiment of the present invention.

[0019] Referring to FIG. 2, the method for learning a spiking neural network based on transformation recognition learning according to the present invention may include an ANN generation step (S110) for generating an analog artificial neural network (ANN) model and inputting variable data, a transformation recognition learning step (S120) for simulating a spiking neural network (SNN) model using one or more activation functions in the ANN model, and an SNN generation step (S130) for generating an SNN model by correcting the parameters and weights of a layer based on the simulation results.

[0020] In this case, analog Artificial Neural Network (ANN) models may include, but are not limited to, Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), and Recurrent Neural Networks (RNN), and are artificial intelligence neural networks excluding Spiking Neural Network (SNN) models.

[0021] And in the transformation recognition learning step (S120), the activation function may be used in one or more layers of the ANN model. At this time, the activation function may include at least one of the ReLU function, the Clip function, and the TTFS (Time to First Spike) function, but is not limited thereto, and is a function that enables SNN simulation.

[0022] In this case, the activation function is a function that transforms the result of the previous layer and transmits a signal to the neurons of another layer, and can improve the complexity of the ANN model. The ReLU function and the Clip function are known activation functions, and the following mathematical formulas can be used.

[0023]

[0024]

[0025] However, the TTFS (Time to First Spike) function is an optimal activation function developed to implement the present invention and can be expressed by the following mathematical formula.

[0026]

[0027] Here, T is time, κ l is the layer kernel, τ is the layer time constant, t l ref θ0 is the start time of the spike, and θ0 is the set threshold value.

[0028] A detailed description of the above activation function will be provided later with reference to FIGS. 3 to 5.

[0029] Additionally, the SNN generation step (S130) can generate the SNN model by transforming the parameters and weights for the layer using at least one of the activation functions.

[0030] FIG. 3 is a flowchart of a transformation recognition learning step according to an embodiment of the present invention, and FIG. 4 is a diagram briefly illustrating a transformation recognition learning-based spiking neural network learning method according to an embodiment of the present invention.

[0031] The transformation recognition learning step (S120) may use the activation function in the layers of the ANN model in the order of ReLU function, Clip function, and TTFS function, but is not limited thereto.

[0032] Referring to FIG. 3, the ReLU function can be used as the first activation function in one or more layers of the ANN model to achieve initial stabilization during training (S210).

[0033] Then, after using the above ReLU function, stable SNN simulation operation can be performed by using the Clip function as the second activation function (S220).

[0034] Then, after using the Clip function, an SNN simulation operation with improved accuracy can be performed using the TTFS function developed to implement the present invention as a third activation function (S230).

[0035] FIGS. 5(a) to 5(c) are graphs of simulation results for a spiking neural network learning method based on transformation recognition learning according to an embodiment of the present invention. FIG. 5(a) is an output graph for data input when the ReLU function is used as the activation function, FIG. 5(b) is an output graph for data input when the Clip function is used as the activation function, and FIG. 5(c) is an output graph for data input when the TTFS function is used as the activation function.

[0036] As shown in Figure 5, it can be seen that the accuracy of the data is improved when activation functions are used in the order of ReLU function, Clip function, and TTFS function.

[0037] Examples 1 to 3 were applied to commercially available datasets, CIFAR10, CIFAR100, and Tiny-ImageNet, respectively. Table 1 is a table comparing the accuracy of the data as a result of training the above datasets on an ANN model.

[0038] Example 1 used only the Clip function as the activation function for each layer of the ANN model, Example 2 used only the TTFS function as the activation function for the first input layer of the ANN model, and Example 3 used the TTFS function as the activation function for all layers of the ANN model.

[0039] experiment T / τ CIFAR10 CIFAR100 Tiny-ImageNet Example 1 48 / 8 92.32(-1.33) 67.93(-4.55) 58.75(-2.28) 24 / 4 86.99(-6.55) 52.48(-20.23) 49.04(-12.03) 12 / 2 62.78(-30.69) 15.07(-57.52) 17.19(-43.84) Example 2 48 / 8 92.85(-0.23) 70.62(-1.06) 59.31(-1.61) 4 / 4 90.92(-1.80) 64.25(-6.34) 51.89(-8.52) 2 / 2 78.21(-12.98) 33.93(-33.27) 21.18(-37.88) Example 3 48 / 8 93.18(-0.02) 71.72(0.00) 60.58(-0.30) 4 / 4 92.45(0.04) 70.30(-0.13) 59.22(-1.05) 2 / 2 90.77(-0.05) 66.00(-0.56) 54.99(-3.90)

[0041] Referring to Table 1, it can be seen that the decrease in accuracy of Example 3 is smaller compared to Examples 1 and 2. Therefore, it can be seen that the accuracy of the data is improved when the TTFS function is used as the activation function in all layers of the ANN model.

[0042] Table 2 is a table comparing the performance of the conventional T2FSNN model (comparative example) and Examples 4 to 6 according to the present invention. At this time, the T2FSNN model of the comparative example is a conventional ANN-to-SNN conversion technique disclosed in the paper "T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding" (authors S. Park, S. Kim, B. Na and S. Yoon).

[0043] Table 2 is a table comparing the performance of Comparative Example and Examples 4 to 6 applied to commercially available datasets CIFAR10, CIFAR100, and Tiny-ImageNet, respectively.

[0044] Looking at the experimental conditions, VGG16 was used as the network, the training length was 200 epochs, the optimizer was SGD (momentum 0.9), and the learning rate was 0.1 (×0.1 on epochs 80, 120, 160).

[0046] Comparative example Example 4 Example 5 Example 6 logarithmic function base e e 2 2 T 80 80 48 24 τ 20 20 8 4 latency 680 1,360 816 408 CIFAR10 91.43 93.36 93.18 92.45 CIFAR100 68.79 72.14 71.72 70.30 Tiny-ImageNet - 60.63 60.58 59.22

[0048] Examples 4 to 6 applied different logarithmic function bases and time conditions as described in Table 2, and used the TTFS function according to the present invention as the activation function.

[0049] When comparing the performance figures of the comparative example and Examples 4 to 6, it can be seen that the performance of the present invention is higher under all conditions. In particular, the T2FSNN model could not be applied to complex datasets such as Tiny-ImageNet.

[0050] Therefore, according to the present invention, high-performance learning that can only be processed by existing ANN learning models can be easily converted and performed using an SNN learning model.

[0051] FIG. 6 is a configuration diagram of a spiking neural network learning system based on transformation recognition learning according to one embodiment of the present invention.

[0052] Referring to FIG. 6, the transformation recognition learning-based spiking neural network learning system (10) according to the present invention may include an ANN generation unit (100) that generates an analog artificial neural network (ANN) model and inputs variable data, a transformation recognition learning unit (200) that simulates a spiking neural network (SNN) model using one or more activation functions in the ANN model, and an SNN generation unit (300) that generates an SNN model by correcting the parameters and weights of a layer based on the simulation results.

[0053] At this time, the transformation recognition learning unit (200) may use the activation function in one or more layers of the ANN model. The activation function may include, but is not limited to, at least one of the ReLU function, the Clip function, and the TTFS (Time to First Spike) function, and is a function that enables SNN simulation. A detailed description of the activation function is as described above.

[0054] In addition, in the transformation recognition learning unit (200), the activation function may be used in the layers of the ANN model in the order of ReLU function, Clip function, and TTFS function, but is not limited thereto.

[0055] And the SNN generation unit (300) can generate the SNN model by converting parameters and weights for the layer using at least one of the activation functions.

[0056] FIG. 7 is a block diagram of a processing device applying a transformation recognition learning-based spiking neural network learning system according to one embodiment of the present invention.

[0057] Referring to FIG. 7, the processing device applying the transformation recognition learning-based spiking neural network learning system according to the present invention can be broadly represented by four components: an input generator, a PE (Process Element) array, an output processing device, and an output control device.

[0058] The above input generator consists of an input buffer of 48KB and a minfind unit and merges input spikes. The above PE array consists of 128 PEs and 4 90KB weight buffers and includes a spiking neural network learning system based on transformation recognition learning according to the present invention.

[0059] The output processing unit is composed of a post-processing unit (PPU) and a spike encoder, processes the output of the PE array into spikes, stores the output spikes in an output buffer, and then transmits the spike information to the DRAM. In addition, the output control unit controls the entire processing unit, and the DMA engine manages data access to the off-chip DRAM.

[0060] At this point, input spikes are processed by aligning them in the input generator, and the aligned spikes are supplied to the PE array and accumulated as membrane potentials. The output of the PE array is transmitted to an output processing unit and encoded as an output spike (ignition stage).

[0061] Therefore, by applying the transformation recognition learning-based spiking neural network learning system according to the present invention to the above-mentioned processing device, data loss occurring during the conversion of ANN training data into SNN training data is minimized.

[0062] Therefore, according to the present invention, high-performance learning that can only be processed by conventional ANN learning models can be performed using an SNN learning model, while enabling hardware operation at low power.

[0063] The above description describes specific embodiments for implementing the present invention. In addition to the embodiments described above, the present invention may also include embodiments that can be simply modified or easily modified. Furthermore, the present invention may also include technologies that can be easily modified and implemented using the embodiments. Accordingly, the scope of the present invention should not be limited to the embodiments described above, but should be defined by the claims set forth below as well as equivalents to the claims of the present invention. Explanation of the symbols

[0064] 10: Spiking Neural Network Learning System 100 : ANN generation unit 200 : Transformation Recognition Learning Unit 300 : SNN generation unit

Claims

Claim 1 The method comprises: an ANN generation step for generating an analog Artificial Neural Network (ANN) model and inputting variable data; a transformation recognition learning step for simulating a Spiking Neural Network (SNN) model using one or more activation functions in the ANN model; and an SNN generation step for generating an SNN model by correcting the parameters and weights of a layer based on the simulation results, wherein the activation function includes a TTFS function defined by the following mathematical formula, and of the following mathematical formula T is time, k l is the layer's kernel, τ is the layer's time constant, t l ref is the start time of the spike, and θ0 is a training method for a spiking neural network based on transformation recognition learning with a set threshold value.[Mathematical Formula] Claim 2 A method for learning a spiking neural network based on transformation recognition learning according to claim 1, wherein the transformation recognition learning step is characterized by using the activation function in one or more layers of the ANN model. Claim 3 A method for training a spiking neural network based on transformation recognition learning, wherein the activation function comprises at least one of a ReLU function, a Clip function, and a TTFS (Time to First Spike) function. Claim 4 delete Claim 5 A method for training a spiking neural network based on transformation recognition learning according to claim 1, characterized in that, in the transformation recognition learning step, the activation functions are used in the ANN model in the order of ReLU function, Clip function, and TTFS function. Claim 6 In claim 1, the SNN generation step is a transformation recognition learning-based spiking neural network learning method that generates the SNN model by transforming parameters and weights for a layer using at least one of the activation functions. Claim 7 An ANN generation unit that generates an analog Artificial Neural Network (ANN) model and inputs variable data; a transformation recognition learning unit that simulates a Spiking Neural Network (SNN) model by using one or more activation functions in the ANN model; and an SNN generation unit that generates an SNN model by correcting the parameters and weights of a layer based on the simulation results, wherein the activation function includes a TTFS function defined by the following mathematical formula, and of the following mathematical formula T is time, k l is the layer's kernel, τ is the layer's time constant, t l ref is the start time of the spike, and θ0 is a spiking neural network learning system based on transformation recognition learning with a set threshold value.[Mathematical Formula] Claim 8 A transform recognition learning-based spiking neural network learning system according to claim 7, wherein the transform recognition learning unit uses the activation function in one or more layers of the ANN model. Claim 9 In claim 7, the activation function is a transformation recognition learning-based spiking neural network learning system comprising at least one of a ReLU function, a Clip function, and a TTFS (Time to First Spike) function. Claim 10 delete Claim 11 A spiking neural network learning system based on transformation recognition learning according to claim 7, characterized in that, in the transformation recognition learning unit, the activation function is applied to the ANN model in the order of ReLU function, Clip function, and TTFS function. Claim 12 In claim 7, the SNN generation unit is a transformation recognition learning-based spiking neural network learning system that generates the SNN model by transforming parameters and weights for the layer using at least one activation function.

Images