A GPU-based spiking neural network simulation method

Abstract

The present invention discloses a pulse neural network simulation method based on GPU, which initializes the neural network structure and network weights; loads the data set and uses the pulse encoding module to perform pulse encoding; calls the GPU calculation module to select the appropriate GPU to calculate the pulse neuron membrane voltage according to the data volume, calculation amount and priority of the calculation task, and compares whether it exceeds the threshold and issues pulses; creates a pulse queue, and adds the neuron that triggers the pulse to the pulse queue; if the pulse queue is not empty, finds the corresponding postsynaptic neuron of the next layer according to the network structure, repeats S3-S5 until the output layer is reached; calculates the loss function according to the result of the output layer and the actual pulse result, and updates the neural network by gradient descent, so that one iteration is completed. The present invention accelerates the training speed of the pulse neural network, gives play to the advantages of the pulse neural network in low power consumption and low latency, and avoids the situation that the data set is too large, the video memory is insufficient, and the training cannot be carried out.

Description

Technical Field

[0001] The present invention belongs to the technical fields of pulse neural networks, brain-like computing and high-performance computing, and specifically relates to a pulse neural network simulation method based on GPU. Background Art

[0002] The spiking neural network is a third-generation artificial neural network, which is a neural network formed by connecting spiking neurons as the basic unit. The membrane potential of spiking neurons accumulates in a nonlinear manner. After reaching the threshold potential, it emits a pulse and enters the refractory period to cool down. At the same time, the pulse is passed to the next neuron. Large spiking neural networks can not only obtain powerful computing power, but also better verify biodynamic characteristics.

[0003] As pulse neural network technology matures, pulse neural networks tend to be deeper and larger networks. The method of using hardware to accelerate pulse neural networks has attracted the attention of researchers. The hardware can be mainly divided into three categories: brain-like chips, heterogeneous architectures combining GPUs and CPUs, and FPGAs. Brain-like chips are tailored for pulse neural networks in terms of power consumption and efficiency, but their popularity is low; FPGAs have the advantages of low power consumption and good flexibility, but they are also not popular, and the learning cost is also a problem; in comparison, using GPUs to accelerate pulse neural networks can greatly improve the research and development efficiency of researchers as GPUs become more and more popular.

[0004] However, the existing GPU-based spiking neural network simulation technology still has the following shortcomings: spiking neural network models are mostly time-driven and do not fully utilize the advantages of pulse sparsity; GPU-based spiking neural networks are mostly accelerated by GPUs and cannot be extended to multiple nodes for acceleration; there are a large number of invalid operations in GPU spiking neural network training, which wastes computing resources. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned technologies, the present invention proposes a GPU-based spiking neural network simulation method, which better utilizes the pulse sparsity characteristics of the spiking neural network by combining time-driven and pulse-driven methods, saving a large amount of invalid calculations; at the same time, it can support large data set training under multiple GPUs and multiple nodes, and can reasonably allocate computing tasks to the GPU according to the task volume and priority.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A GPU-based spiking neural network simulation method comprises the following steps:

[0008] S1. Initialize the neural network structure and network weights;

[0009] S2. Load the data set and perform pulse encoding using the pulse encoding module;

[0010] S3. Call the GPU computing module to select a suitable GPU to calculate the pulse neuron membrane voltage according to the data volume, computing volume and priority of the computing task, and compare whether it exceeds the threshold and issue a pulse;

[0011] S4. Create a pulse queue and add the neuron that triggers the pulse to the pulse queue;

[0012] S5. If the pulse queue is not empty, find the corresponding postsynaptic neuron in the next layer according to the network structure, and repeat steps S3-S5 until the output layer is reached;

[0013] S6. Calculate the loss function based on the results of the output layer and the actual pulse results, and use the gradient descent method to update the neural network. At this point, one iteration is completed.

[0014] The specific steps of step S1 are:

[0015] S11. Select a suitable neural network structure according to the type of data set and build the corresponding neural network structure; the network weights are randomly generated using random numbers.

[0016] Furthermore, the specific steps of step S2 are:

[0017] S21. The pulse encoding module adopts the first pulse encoding method, that is, the pulse emission time is calculated according to the maximum and minimum values ​​of the input data set; first, the total time slice length is set to T, and then the unique pulse emission time is calculated according to the maximum, minimum and current values ​​of each feature of the data set. The formula is:

[0018]

[0019] Where: t i It indicates the time of pulse emission after the i-th data is converted into pulse code, I max Represents the maximum value of each feature of the data set, I min Represents the minimum value of each feature of the data set, I i represents the current value of each feature of the data set, and T represents the total time slice length; after the data set is pulse-encoded by the pulse coding module, each real number of the data set is converted into a pulse sequence containing only 0 and 1, and the data set is processed into a pulse sequence;

[0020] Furthermore, the specific steps of step S3 are:

[0021] S31. Call the GPU calculation module to obtain the GPU information of the current computer. The calculation module will sort the GPUs according to the GPU memory usage and GPU utilization rate, and give priority to calling the GPUs that are idle and have remaining memory to process urgent tasks. The specific GPU selection is also determined by the data volume, calculation amount and priority of the calculation task itself;

[0022] S32. The computing task is denoted as τ. There are n computing tasks in total, where the i-th task is denoted as τ i , assuming that the task queue Γ = {τ 0 , τ 1 ,…τ i ,…τ n-1 There are n tasks, each of which has three attributes: data volume, computational load, and priority. The data volume of a task is calculated based on the size of the data set to determine its memory usage. The computational load of a task is calculated based on the number of actual computational operations of the task multiplied by the size of the data volume. The priority level satisfies that the priority of the back-layer network is higher than that of the front-layer network, and that the priority of a task with an earlier time slice is higher than that of a task with a later time slice.

[0023] S33. When a single task needs to be processed, first find out whether there is a GPU that meets the requirements according to the three attributes of the task, find the most idle GPU, and start processing the task; when no GPU that meets the requirements is found, if the amount of data cannot be loaded into the video memory at one time, call the function of the calculation module to split the data;

[0024] S34. When multiple tasks need to be processed at the same time, first select the high-priority task according to the priority of the task and choose the appropriate GPU to complete the computing needs. When the priorities are the same, the task with a smaller data volume is given priority to complete. When the data volumes are also the same, the task with a smaller computing volume is given priority to complete.

[0025] S35. Finally, the voltage value of the pulse neuron calculated according to the task is compared with the threshold to determine whether to send a pulse. Let τ m is called the membrane time constant, R m is the resistance constant, u(t) is the membrane voltage value, I(t) is the input current, and the LIF neuron membrane voltage calculation formula can be written as follows:

[0026]

[0027] The voltage value of a specific LIF neuron is determined by the current I(t), and the input current I(t) is calculated from the input pulses on each synapse and their weights. Let N be the number of neuron input synapses, w j is the weight of the jth synapse of neuron, I j(t) is the pulse generated by the jth synapse at time t, and the input current I(t) is written as follows:

[0028]

[0029] Furthermore, the specific steps of step S4 are:

[0030] S41. The pulse queue is a container for storing neurons that emit pulses at the current moment. When the model starts iterating from moment t=0, it first calculates the voltage value of the input pulse according to step S3 to find the neuron that generates the pulse. If no pulse is generated, it directly jumps to the next moment to re-input the pulse; if a pulse is generated, it finds the corresponding connected neuron in the next layer of the neural network, and puts the corresponding connected neuron in the next layer into the queue, and adjusts the voltage of the neuron that emits the pulse to the reset voltage.

[0031] Furthermore, the specific steps of step S5 are:

[0032] S51. If the pulse queue is not empty, find the corresponding postsynaptic neuron in the next layer according to the network structure, repeat steps S3-S5 until the output layer is reached and the current moment is the total length of the time slice T, and finally save its output pulse sequence; when the neuron in the pulse queue is an output layer neuron and the current moment is not T, then t=t+1 starts iterating from the input pulse.

[0033] Furthermore, the specific steps of step S6 are:

[0034] S61. Calculate the loss function based on the error between the output layer result and the actual pulse result. Let e ​​be the loss list of all output neurons, c be the number of output neurons, and e k is the loss of the kth output neuron, T k is the pulse emission time of the kth output neuron in the target pulse output sequence, t k is the pulse emission time of the kth output neuron in the actual pulse output sequence, which is specifically defined as follows:

[0035] e=[e 1 , e 2 ,…,e c ]and e k =(T k -t k ) / T

[0036] The total loss function L is defined as:

[0037]

[0038] S62. Update the neural network using a gradient descent method according to the calculated loss function;

[0039] For the hidden layer, the Spiking Relu function is used to update the weights. is the m neurons in the lth layer, is the connection weight between the mth neuron in the lth layer and the pth neuron in the previous layer, Whether the p neurons in the l-1 layer emit pulses, the update rule is as follows:

[0040]

[0041] The update rule of the LIF model is as follows, L is the total loss function, is the connection weight between the mth neuron in the lth layer and the pth neuron in the previous layer, is the loss value of the mth neuron in the lth layer, is the pulse emission time of the mth neuron in the lth layer, is the pulse train of the pth neuron in the l-1th layer.

[0042]

[0043] Batch backpropagation updates the neural network weights until this iteration is complete.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention uses GPU to simulate and accelerate the pulse neural network, and proposes a pulse neural network that combines pulse drive and time drive and is realized by pulse queue mode. In addition, the present invention proposes a strategy for task distribution of pulse neural network under multiple GPUs. By distributing the computing tasks to each GPU according to the strategy through pulse drive mode, the effect of multi-GPU accelerated computing can be exerted, and a large number of invalid operations without pulses can be avoided, which greatly reduces the time cost of model training and gives full play to the advantages of pulse neural network in low power consumption and low latency.

[0046] The pulse drive and time drive combination strategy and multi-GPU task allocation strategy involved in the present invention are highly versatile and scalable, and can be applied to most pulse neural network situations to make up for the disadvantages of pulse neural network in training. At the same time, due to its advantages of low latency and low energy consumption, it can be more conveniently applied to actual production life. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0048] Figure 2 Design diagrams for software architecture;

[0049] Figure 3 It is the system flow chart;

[0050] Figure 4 It is a time-driven and event-driven simulation method;

[0051] Figure 5 Strategy for distributing tasks to GPUs. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the embodiments.

[0053] Example 1

[0054] A GPU-based spiking neural network simulation method comprises the following steps:

[0055] S1. Initialize the neural network structure and network weights;

[0056] The specific steps of step S1 are:

[0057] S11. Select the appropriate neural network structure according to the type of data set, and build the corresponding neural network structure. Specifically: for example, for the MNIST data set, you can choose to build a two-layer neural network structure, and use a fully connected method to connect the neurons between layers; for the CiFar data set, you can build a multi-layer neural network structure, first perform some convolutional layer operations, and then output the results through the fully connected layer; for the Iris data set, you can use a single neuron structure; the network weights are randomly generated using random numbers, that is, the network weights are randomly generated using random numbers with a certain range of normal distribution weights.

[0058] S2. Load the data set and perform pulse encoding using the pulse encoding module;

[0059] The specific steps of step S2 are:

[0060] S21. The pulse encoding module adopts the first pulse encoding method, that is, the pulse emission time is calculated according to the maximum and minimum values ​​of the input data set; first, the total time slice length is set to T, and then the unique pulse emission time is calculated according to the maximum, minimum and current values ​​of each feature of the data set. The formula is:

[0061]

[0062] Where: t i It indicates the time of pulse emission after the i-th data is converted into pulse code, I max Represents the maximum value of each feature of the data set, I min Represents the minimum value of each feature of the data set, I irepresents the current value of each feature of the data set, and T represents the total time slice length; after the data set is pulse-encoded by the pulse coding module, each real number of the data set is converted into a pulse sequence containing only 0 and 1, and the data set is processed into a pulse sequence;

[0063] S3. Call the GPU computing module to select a suitable GPU to calculate the pulse neuron membrane voltage according to the data volume, computing volume and priority of the computing task, and compare whether it exceeds the threshold and issue a pulse;

[0064] The specific steps of step S3 are:

[0065] S31. Call the GPU calculation module to obtain the GPU information of the current computer. The calculation module will sort the GPUs according to the GPU memory usage and GPU utilization rate, and give priority to calling the GPUs that are idle and have remaining memory to process urgent tasks. The specific GPU selection is also determined by the data volume, calculation amount and priority of the calculation task itself;

[0066] S32. The computing task is denoted as τ. There are n computing tasks in total, where the i-th task is denoted as τ i , assuming that the task queue Γ = {τ 0 , τ 1 ,…τ i ,…τ n - 1 There are n tasks, each of which has three attributes: data volume, computational load, and priority. The data volume of a task is calculated based on the size of the data set to determine its memory usage. The computational load of a task is calculated based on the number of actual computational operations of the task multiplied by the size of the data volume. The priority level satisfies that the priority of the back-layer network is higher than that of the front-layer network, and that the priority of a task with an earlier time slice is higher than that of a task with a later time slice.

[0067] S33. When a single task needs to be processed, first find out whether there is a GPU that meets the requirements according to the three attributes of the task, find the most idle GPU, and start processing the task; when no GPU that meets the requirements is found, if the amount of data is too large, that is, the amount of data cannot be loaded into the video memory at one time, then call the function function of the calculation module to split the data;

[0068] S34. When multiple tasks need to be processed at the same time, first select the high-priority task according to the priority of the task and choose the appropriate GPU to complete the computing needs. When the priorities are the same, the task with a smaller data volume is given priority to complete. When the data volumes are also the same, the task with a smaller computing volume is given priority to complete.

[0069] S35. Finally, the voltage value of the pulse neuron calculated according to the task is compared with the threshold to determine whether to send a pulse. Let τ mis called the membrane time constant, R m is the resistance constant, u(t) is the membrane voltage value, I(t) is the input current, and the LIF neuron membrane voltage calculation formula can be written as follows:

[0070]

[0071] The voltage value of a specific LIF neuron is determined by the current I(t), and the input current I(t) is calculated from the input pulses on each synapse and their weights. Let N be the number of neuron input synapses, w j is the weight of the jth synapse of neuron, I j (t) is the pulse generated by the jth synapse at time t, and the input current I(t) is written as follows:

[0072]

[0073] S4. Create a pulse queue and add the neuron that triggers the pulse to the pulse queue;

[0074] The specific steps of step S4 are:

[0075] S41. The pulse queue is a container for storing neurons that emit pulses at the current moment. When the model starts iterating from moment t=0, it first calculates the voltage value of the input pulse according to step S3 to find the neuron that generates the pulse. If no pulse is generated, it directly jumps to the next moment to re-input the pulse; if a pulse is generated, it finds the corresponding connected neuron in the next layer of the neural network, and puts the corresponding connected neuron in the next layer into the queue, and adjusts the voltage of the neuron that emits the pulse to the reset voltage.

[0076] S5. If the pulse queue is not empty, find the corresponding postsynaptic neuron in the next layer according to the network structure, and repeat steps S3-S5 until the output layer is reached;

[0077] The specific steps of step S5 are:

[0078] S51. If the pulse queue is not empty, find the corresponding postsynaptic neuron in the next layer according to the network structure, repeat steps S3-S5 until the output layer is reached and the current moment is the total length of the time slice T, and finally save its output pulse sequence; when the neuron in the pulse queue is an output layer neuron and the current moment is not T, then t=t+1 starts iterating from the input pulse.

[0079] S6. Calculate the loss function based on the result of the output layer and the actual pulse result, and update the neural network using the gradient descent method. At this point, one iteration is completed;

[0080] The specific steps of step S6 are:

[0081] S61. Calculate the loss function based on the error between the output layer result and the actual pulse result. Let e ​​be the loss list of all output neurons, c be the number of output neurons, and e k is the loss of the kth output neuron, T k is the pulse emission time of the kth output neuron in the target pulse output sequence, t k is the pulse emission time of the kth output neuron in the actual pulse output sequence, which is specifically defined as follows:

[0082] e=[e 1 , e 2 ,…,e c ]and e k =(T k -t k ) / T

[0083] The total loss function L is defined as:

[0084]

[0085] S62. Update the neural network using a gradient descent method according to the calculated loss function;

[0086] For the hidden layer, the Spiking Relu function is used to update the weights. is the m neurons in the lth layer, is the connection weight between the mth neuron in the lth layer and the pth neuron in the previous layer, Whether the p neurons in the l-1 layer emit pulses, the update rule is as follows:

[0087]

[0088] The update rule of the LIF model is as follows, L is the total loss function, is the connection weight between the mth neuron in the lth layer and the pth neuron in the previous layer, is the loss value of the mth neuron in the lth layer, is the pulse emission time of the mth neuron in the lth layer, is the pulse train of the pth neuron in the l-1th layer.

[0089]

[0090] Batch backpropagation updates the neural network weights until this iteration is complete.

[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A GPU-based spiking neural network simulation method, It is characterized in that The following steps are involved: S1. Initialize the neural network structure and network weights; S2. Load the data set and perform pulse encoding using the pulse encoding module; S3. Call the GPU computing module to select a suitable GPU to calculate the pulse neuron membrane voltage according to the data volume, computing volume and priority of the computing task, and compare whether it exceeds the threshold and issue a pulse; S4. Create a pulse queue and add the neuron that triggers the pulse to the pulse queue; S5. If the pulse queue is not empty, find the corresponding postsynaptic neuron in the next layer according to the network structure, and repeat steps S3-S5 until the output layer is reached; S6. Calculate the loss function based on the result of the output layer and the actual pulse result, and update the neural network using the gradient descent method. At this point, one iteration is completed; The specific steps of step S2 are: S21. The pulse encoding module adopts the first pulse encoding method, that is, the pulse emission time is calculated according to the maximum and minimum values ​​of the input data set; first, the total time slice length is set to T, and then the unique pulse emission time is calculated according to the maximum, minimum and current values ​​of each feature of the data set. The formula is: Where: t i It indicates the time of pulse emission after the i-th data is converted into pulse code, I max Represents the maximum value of each feature of the data set, I min Represents the minimum value of each feature of the data set, I i represents the current value of each feature of the data set, and T represents the total time slice length; after the data set is pulse-encoded by the pulse coding module, each real number of the data set is converted into a pulse sequence containing only 0 and 1, and the data set is processed into a pulse sequence; The specific steps of step S3 are: S31. Call the GPU calculation module to obtain the GPU information of the current computer. The calculation module will sort the GPUs according to the GPU memory usage and GPU utilization rate, and give priority to calling the GPUs that are idle and have remaining memory to process urgent tasks. The specific GPU selection is also determined by the data volume, calculation amount and priority of the calculation task itself; S32. The computing task is denoted as τ. There are n computing tasks in total, where the i-th task is denoted as τ i , assuming that the task queue Γ = {τ 0 ,τ 1 ,…τ i ,…τ n-1 There are n tasks, each of which has three attributes: data volume, computational load, and priority. The data volume of a task is calculated based on the size of the data set to determine its memory usage. The computational load of a task is calculated based on the number of actual computational operations of the task multiplied by the size of the data volume. The priority level satisfies that the priority of the back-layer network is higher than that of the front-layer network, and that the priority of a task with an earlier time slice is higher than that of a task with a later time slice. S33. When a single task needs to be processed, first find out whether there is a GPU that meets the requirements according to the three attributes of the task, find the most idle GPU, and start processing the task; when no GPU that meets the requirements is found, if the amount of data cannot be loaded into the video memory at one time, call the function of the calculation module to split the data; S34. When multiple tasks need to be processed at the same time, first select the high-priority task according to the priority of the task and choose the appropriate GPU to complete the computing needs. When the priorities are the same, the task with a smaller data volume is given priority to complete. When the data volumes are also the same, the task with a smaller computing volume is given priority to complete. S35. Finally, the voltage value of the pulse neuron calculated according to the task is compared with the threshold to determine whether to send a pulse. Let τ m is called the membrane time constant, R m is the resistance constant, u(t) is the membrane voltage value, I(t) is the input current, and the LIF neuron membrane voltage calculation formula is written as follows: The voltage value of a specific LIF neuron is determined by the current I(t), and the input current I(t) is calculated from the input pulses on each synapse and their weights. Let N be the number of neuron input synapses, w j is the weight of the jth synapse of neuron, I j (t) is the pulse generated by the jth synapse at time t, and the input current I(t) is written as follows: The specific steps of step S4 are: S41. The pulse queue is a container for storing neurons that emit pulses at the current moment. When the model starts iterating from moment t=0, it first calculates the voltage value of the input pulse according to step S3 to find the neuron that generates the pulse. If no pulse is generated, it directly jumps to the next moment to re-input the pulse; if a pulse is generated, it finds the corresponding connected neuron in the next layer of the neural network, and puts the corresponding connected neuron in the next layer into the queue, and adjusts the voltage of the neuron that emits the pulse to the reset voltage.

2. The GPU-based spiking neural network simulation method according to claim 1, It is characterized in that The specific steps of step S1 are: S11. Select a suitable neural network structure according to the type of data set and build the corresponding neural network structure; the network weights are randomly generated using random numbers.

3. The GPU-based spiking neural network simulation method according to claim 1, It is characterized in that The specific steps of step S5 are: S51. If the pulse queue is not empty, find the corresponding postsynaptic neuron in the next layer according to the network structure, repeat steps S3-S5 until the output layer is reached and the current moment is the total length of the time slice T, and finally save its output pulse sequence; when the neuron in the pulse queue is an output layer neuron and the current moment is not T, then t=t+1 starts iterating from the input pulse.

4. The GPU-based spiking neural network simulation method according to claim 1, It is characterized in that The specific steps of step S6 are: S61. Calculate the loss function based on the error between the output layer result and the actual pulse result. Let e ​​be the loss list of all output neurons, c be the number of output neurons, and e k is the loss of the kth output neuron, T k is the pulse emission time of the kth output neuron in the target pulse output sequence, t k is the pulse emission time of the kth output neuron in the actual pulse output sequence, which is specifically defined as follows: e=[e 1 ,e 2 ,…,e c ] and e k =(T k -t k ) / T The total loss function L is defined as: S62. Update the neural network using a gradient descent method according to the calculated loss function; For the hidden layer, the Spiking Relu function is used to update the weights. is the m neurons in the lth layer, is the connection weight between the mth neuron in the lth layer and the pth neuron in the previous layer, Whether the p neurons in the l-1 layer emit pulses, the update rule is as follows: The update rule of the LIF model is as follows, L is the total loss function, is the connection weight between the mth neuron in the lth layer and the pth neuron in the previous layer, is the loss value of the mth neuron in the lth layer, is the pulse emission time of the mth neuron in the lth layer, is the pulse train of the pth neuron in the l-1th layer; Batch backpropagation updates the neural network weights until this iteration is complete.

Images