A computing system for spiking recurrent neural networks

Abstract

The present application discloses a computing system for a pulse recurrent neural network, comprising: a preprocessing module for encoding different types of target data into 8-bit target pulse data; a core computing module for accelerating parallel computing of the target pulse data through a dual-core neuron group, selecting the number of neurons, neuron types and weights according to the computing task, the dual-core neuron group including two neuron cores, and the neuron core including two groups of sub-neuron groups in parallel; a storage module for storing pulse signals, membrane potential information and weights adapted to neurons during the computing process of the core computing module; a control module for controlling the preprocessing module to process data, controlling the core computing module to select neurons and trigger bidirectional computing, and controlling the storage module to store data. The present application can solve the technical problems that the existing technology does not support the deployment of complex networks and bidirectional networks, and the simultaneous processing of dual tasks and dual samples, resulting in limited actual deployment operations and low efficiency.

Description

Technical Field

[0001] The present application relates to the field of neural network deployment technology, and in particular to a computing system for pulse recurrent neural networks. Background Art

[0002] Spiking Neural Networks (SNNs) are third-generation neural networks that use spiking neurons as computational units and transmit information through synaptic connections within the network, employing discrete neuronal spikes. By simulating the information processing mechanisms of the biological brain, SNNs aim to build artificial intelligence systems with superior computational capabilities and closely tied to biological neuroscience principles. Implementing SNNs on hardware platforms primarily involves the computational implementation of neuronal dynamics models and the design of efficient information transmission structures, both of which are core elements of neuromorphic hardware design. SNNs have many fundamental computational units, including the Hodgkin-Huxley (HH) model, the Izhikevich (IZH) model, the Integred-and-Fire (IF) model, and the Leaky Integred-and-Fire (LIF) model.

[0003] Spiking Recurrent Neural Networks (SRNNs) and Bidirectional Spiking Recurrent Neural Networks (Bi-SRNNs) are both evolved from SNNs. These evolved networks offer significant advantages in data processing and feature analysis. However, the recurrent structure of SRNNs and the bidirectional processing requirements of Bi-SRNNs increase the complexity and difficulty of hardware design.

[0004] Specifically, most existing technologies can only meet the needs of relatively shallow network deployments and cannot be deployed for multi-layer complex recurrent neural networks. Moreover, most deployments are for unidirectional SRNNs and do not support Bi-SRNNs. In addition, the deployed systems or platforms do not support the simultaneous processing of dual tasks and dual samples, so the data processing efficiency is also very low, which makes it difficult to meet the needs of actual application scenarios. Summary of the Invention

[0005] The present application provides a computing system for pulse recurrent neural networks, which is used to solve the technical problems that the existing technology does not support the deployment of multi-layer complex networks and bidirectional networks, nor does it support the simultaneous processing of dual tasks and dual samples, resulting in limited actual deployment operations and low efficiency.

[0006] In view of this, the present application provides a computing system for a pulse recurrent neural network, comprising: a preprocessing module, a core computing module, a storage module and a control module;

[0007] The preprocessing module is used to encode different types of target data into 8-bit target pulse data, wherein the target data includes image data, sequence data and text data;

[0008] The core computing module is used to accelerate parallel computing of the target pulse data through a deployed dual-core neuron group, and select the number of neurons, neuron types and weights according to the computing task. The dual-core neuron group includes two neuron cores, each of which includes two groups of sub-neuron groups connected in parallel, and the neuron types include IF neurons and improved LIF neurons;

[0009] The storage module is used to store the pulse signals, membrane potential information and weights adapted to neurons generated by the core computing module during the computing process;

[0010] The control module is used to control the preprocessing module to process data, control the core computing module to select neurons and trigger bidirectional computing, and control the storage module to store data.

[0011] Preferably, the pre-processing module includes:

[0012] An encoding submodule, configured to encode different target data into initial pulse data using different preset encoding algorithms, wherein the preset encoding algorithms include binary encoding, BSA encoding, and rate encoding;

[0013] The integration submodule is used to compress the pulse signal and merge it into 8-bit real number data to obtain target pulse data.

[0014] Preferably, it also includes: a data transmission module;

[0015] The data transmission module is used to transmit the target pulse data to the FPGA for data calculation through a preset transmission technology, and the preset transmission technology includes a UART interface.

[0016] Preferably, it also includes: a data sorting module;

[0017] The data sorting module is used to perform splicing and sorting processing on the target pulse data to obtain rearranged pulse data, and transmit the rearranged pulse data to the core calculation module for data calculation. The rearranged pulse data includes a pulse index.

[0018] Preferably, the core computing module is specifically used to:

[0019] Reading the rearranged pulse data from a buffer;

[0020] Selecting the number and type of neurons according to the computational task, and selecting weights corresponding to the neurons based on the pulse index;

[0021] The target pulse data is accelerated and parallelized by the deployed dual-core neuron group to obtain a data calculation result, which includes the result pulse data and the result membrane potential.

[0022] Preferably, the process of obtaining the improved LIF neurons is:

[0023] Based on the LIF neuron calculation principle, the original LIF dynamic equation is discretized into a simplified equation to obtain an improved LIF neuron. The simplified equation is expressed as:

[0024]

[0025] in, is the membrane potential decay rate constant, for When improving the output membrane potential of LIF neurons, for The membrane potential at for The neuron input data at time .

[0026] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0027] In the present application, a computing system for a pulse recurrent neural network is provided, including: a preprocessing module, a core computing module, a storage module and a control module; the preprocessing module is used to encode different types of target data into 8-bit target pulse data, and the target data includes image data, sequence data and text data; the core computing module is used to accelerate the parallel calculation of the target pulse data through the deployed dual-core neuron group, and the number of neurons, neuron types and weights are selected according to the calculation task, the dual-core neuron group includes two neuron cores, each neuron core includes two groups of sub-neuron groups in parallel, and the neuron types include IF neurons and improved LIF neurons; the storage module is used to store the pulse signals, membrane potential information and weights adapted to the neurons generated by the core computing module during the calculation process; the control module is used to control the preprocessing module to perform data processing, control the core computing module to select neurons and trigger bidirectional calculations, and control the storage module to store data.

[0028] The computing system for pulse recurrent neural networks provided by this application provides a deployment platform for various forms of pulse recurrent neural networks, supports data processing and calculation of different forms of pulse recurrent neural networks; provides a basis for accelerated parallel computing by designing a dual-core neuron group, and optimizes the computing process by improving the LIF neuron; selects a unidirectional or bidirectional computing strategy based on the control module; this process supports parallel computing of complex networks, meets the processing requirements of bidirectional networks, and can also process dual samples of a single task and dual tasks at the same time; in addition, for network architectures that require buffering functions to manage network parameters and parallel operations, this requirement can also be met here through a storage module, which greatly improves the compatibility and adaptability of the deployment platform and can meet the deployment requirements of actual scenarios. Therefore, this application can solve the technical problem that the existing technology does not support the deployment of multi-layer complex networks and bidirectional networks, nor does it support the simultaneous processing of dual tasks and dual samples, resulting in limited actual deployment operations and low efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the structure of a computing system for a spiking recurrent neural network provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the internal data flow processing structure of the data sorting module provided in an embodiment of the present application;

[0031] Figure 3 Schematic diagram of the hardware implementation of the LIF / IF neuron provided in the embodiment of the present application;

[0032] Figure 4 A schematic diagram of a flow chart of a control module controlling core calculations provided in an embodiment of the present application;

[0033] Figure 5 Schematic diagram of the overall framework of the computing system for spiking recurrent neural networks provided in an embodiment of the present application;

[0034] Figure 6 Schematic diagram of the hardware structure involved in data stream processing under different topological structures and different types of neural network deployments provided in the embodiments of this application. DETAILED DESCRIPTION

[0035] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0036] For easier understanding, see Figure 1 , an embodiment of a computing system for a pulse recurrent neural network provided in this application includes: a preprocessing module 101, a core computing module 102, a storage module 103 and a control module 104.

[0037] The pre-processing module 101 is used to encode different types of target data into 8-bit target pulse data, where the target data includes image data, sequence data and text data.

[0038] Furthermore, the pre-processing module 101 includes:

[0039] The encoding submodule is used to encode different target data into initial pulse data using different preset encoding algorithms, where the preset encoding algorithms include binary encoding, BSA encoding, and rate encoding;

[0040] The integration submodule is used to compress the pulse signal and merge it into 8-bit real number data to obtain the target pulse data.

[0041] It should be noted that the data preprocessing process is an operation completed on the PC side. It requires encoding different types of data into pulse data, and then compressing and integrating them into a format that meets the transmission requirements and neural network processing requirements. For data encoding, binary encoding can be used to encode simple image data, frequency encoding can be used to encode complex image data, and BSA encoding can be used to encode sequence data. This is only an example and not a limitation. You can choose the appropriate encoding algorithm for different types of data according to actual needs. Therefore, the preset encoding algorithms include but are not limited to binary encoding, BSA encoding, and rate encoding. Similarly, the target data includes but is not limited to image data, sequence data, and text data.

[0042] In this embodiment, the encoded initial pulse data is compressed and combined to produce 8-bit real-number data, the target pulse data. This data can then be transmitted to the FPGA accelerated computing platform for data calculation. Encoding and then compressing the data can reduce data transmission time to one-eighth the original time, improving data processing efficiency and reducing system runtime.

[0043] The core computing module 102 is used to accelerate the parallel computing of the target pulse data through the deployed dual-core neuron group, and select the number of neurons, neuron types and weights according to the computing task. The dual-core neuron group includes two neuron cores, each neuron core includes two groups of sub-neuron groups in parallel, and the neuron types include IF neurons and improved LIF neurons.

[0044] The storage module 103 is used to store the pulse signals, membrane potential information and weights adapted to neurons generated by the core computing module during the computing process.

[0045] The control module 104 is used to control the pre-processing module to process data, control the core computing module to select neurons and trigger bidirectional computing, and control the storage module to store data.

[0046] Furthermore, it also includes: a data transmission module;

[0047] The data transmission module is used to transmit the target pulse data to the FPGA for data calculation through a preset transmission technology, and the preset transmission technology includes a UART interface.

[0048] Furthermore, it also includes: a data sorting module;

[0049] The data sorting module is used to perform splicing and sorting processing on the target pulse data to obtain rearranged pulse data, and transmit the rearranged pulse data to the core calculation module for data calculation. The rearranged pulse data includes a pulse index.

[0050] Furthermore, the core computing module 102 is specifically configured to:

[0051] Reading rearranged pulse data from the buffer;

[0052] The number and type of neurons are selected based on the computational task, and the weights corresponding to the neurons are selected based on the pulse index.

[0053] The target pulse data is accelerated and parallelized by the deployed dual-core neuron group to obtain data calculation results, which include result pulse data and result membrane potential.

[0054] Furthermore, the acquisition process of LIF neurons is improved as follows:

[0055] Based on the calculation principle of LIF neurons, the original LIF dynamic equation is discretized into a simplified equation to obtain the improved LIF neuron. The simplified equation is expressed as:

[0056]

[0057] in, is the membrane potential decay rate constant, for When improving the output membrane potential of LIF neurons, for The membrane potential at for The neuron input data at time .

[0058] It should be noted that the computing core on the FPGA acceleration platform of this embodiment is a dual-core neuron group consisting of two identical neuron cores, i.e., a dual-core; each neuron core includes two parallel sub-neuron groups, i.e., the dual-core neuron group includes four sub-neuron groups, and each sub-neuron group includes multiple neurons, which are also parallel. Specifically, in this embodiment, each sub-neuron group includes 64 parallel neurons, and two sub-neuron groups are connected in parallel to form a neuron core, i.e., the two neuron cores have a total of 256 neurons. The use of multiplexing and parallel four-stage pipeline technology can support inference calculations of various different topologies.

[0059] See also Figure 2 After the target pulse data enters the FPGA, it must first be reordered to ensure that different types of task data can be efficiently processed by the corresponding neurons. For example, reordering operations can be performed to splice cyclic data, splice and sort bidirectional data, and sort multi-task data streams and single-task multi-sample data. Reordering operations can organize complex data streams into orderly data streams, speeding up the system's computing speed and improving data processing efficiency.

[0060] The obtained rearranged pulse data is the data that the input neurons actually participate in the calculation, and the rearranged pulse data includes the corresponding pulse index. Specifically, after reading the rearranged pulse data and the corresponding pulse index in the buffer, you can first select the number of parallel neurons required for the current calculation task, such as 64, 128, 192 and 256, etc. These neurons can accelerate the parallel processing of the current task data. In addition, it is also necessary to select a specific type of neuron. The types of neurons in this embodiment mainly include IF neurons and LIF neurons, and LIF neurons are improved neurons. The working principles of the two types of neurons can be found in Figure 3 The neuron calculation in this embodiment is event-driven calculation, that is, the calculation is started only when the neuron emits a pulse.

[0061] After reading the rearranged pulse data, the neuron will traverse the source neuron address stored in the buffer FIFO according to the pulse index, and obtain the weight corresponding to the neuron according to the address. Then the neuron can perform calculations based on the extracted weight; a one-to-many parallel computing method can be used to update the status of 64 neurons at a time to speed up the calculation process.

[0062] This embodiment selects different types of neurons for calculation during the calculation process. The specific processes are different. Please refer to Figure 4If you choose to use an improved LIF neuron, the specific calculation process is as follows: ①②③④. If you choose an IF neuron, the membrane potential leakage operation (step ④) is not required; the calculation process is as follows: ①②③. If you deploy a fully connected network, during the control process, the membrane potential is stored directly after storing the pulse signal. If you deploy a spiking recurrent neural network, the membrane potential needs to be stored after storing the pulse signal.

[0063] The improved LIF neuron of this embodiment is the result of optimization based on the computational principle of neurons. The computation of LIF neurons mainly includes three processes, namely leakage, integration, and emission, which can be expressed by a specific dynamic equation:

[0064]

[0065]

[0066] in, is the time constant, is the membrane potential voltage, is the neuron reset voltage, is the neuron threshold voltage, For time, is the neuron input. By using continuous time discretization and differential calculation to approximate the derivative, the above equation is solved. Since the leakage of membrane potential is in terms of voltage, the neuron input as the external applied signal is No need to divide by the time constant , which is independent of the intrinsic leakage process of the membrane potential. So we can get the following formula:

[0067]

[0068] Assume that a time step is expressed as , then at discrete time points ,in, Is an integer, we can use the difference to calculate the approximate derivative, then we can get the following formula:

[0069]

[0070] Rearranging the above formulas, we can get:

[0071]

[0072] In the formula Extract it and you get:

[0073]

[0074] definition , used to describe the membrane potential decay rate, is a constant that can be set. , , and the time step of the input neuron signal is considered to be is constant, then based on the above formula we can get the simplified equation expression of the improved LIF neuron:

[0075]

[0076] If the membrane potential decay rate constant Set to , then the shift method can be used to implement division on the hardware platform, and the specific calculation formula can be obtained:

[0077]

[0078] The neuron membrane potential decay calculation can be completed by subtracting the result of the right shift by three digits from the original membrane potential. When the neuron membrane potential reaches the threshold, a pulse will be emitted. The emitted pulse consumes the accumulated charge of the neuron, so the membrane potential will be reset. In order to simplify the behavior of the neuron, this embodiment directly sets the reset potential to 0, that is, See Figure 3 In ①②, if it is an IF neuron, there is no need to decay the membrane potential, only the input needs to be accumulated, that is:

[0079]

[0080] The improved LIF neuron of this embodiment selects whether to perform neuron integral calculation according to whether the input pulse is 1. Figure 3 ① in step 1 satisfies the event-driven nature of LIF neurons. Pulse emission is then achieved based on the comparator in step ②, and finally, the membrane potential is leaked through step ③. This embodiment uses a shifter and a subtractor to implement leakage calculation by multiplying the membrane potential by a decay constant. The actual hardware requirements are simple, making the solution easy to implement.

[0081] During the specific calculation process of the core computing module 102 and after the calculation is completed, some pulse data and membrane potential will be generated. These may be generated after each network layer calls the calculation core, or they may be generated by the network layer itself. These signals can be cached or stored in the memory or buffer, and the weights are also pre-cached in the memory to support the network to achieve efficient calculation. In the deployment process of different spiking neural networks, pulse index storage and cyclic pulse storage may also be required. These can be achieved based on the control trigger of the control module and will not be repeated here.

[0082] The control module of this embodiment can be divided into top-level control and computational control according to the control function; top-level control is the control of data preparation and data caching, such as controlling the preprocessing module to process data; top-level control can realize the control of the overall network architecture by controlling the data process, thereby realizing the accelerated calculation of networks with different topological structures. The computational control part is the selection of neurons, which includes the selection of the number and type of neurons; in addition, the system can also be controlled to perform bidirectional calculations, which can meet the deployment and calculation of bidirectional pulse recurrent neural networks. It can also control the storage module to cache or store the pulse and membrane potential information of some deployed networks; the control module can obtain different accelerated networks by interacting with different modules; please refer to the overall framework of the computing system designed in this embodiment. Figure 5 .

[0083] In this embodiment, the data preprocessing and data calculation processes are located on the PC and FPGA, respectively. Communication between the two is achieved through a pre-configured transmission technology to ensure efficient and reliable data transmission. This transmission involves transmitting target pulse data to the FPGA and transmitting the calculated result data back to the PC. For example, a UART interface can be used; other communication devices or devices can also be used depending on the actual situation. This is provided for illustrative purposes only and is not intended to be limiting.

[0084] It should be noted that the computing system for pulse recurrent neural networks involved in this embodiment can decide whether to use different data processing data flows according to different pulse neural network models. If a fully connected network is deployed, the data sorting process only needs to process the input coded pulses and the output pulses of each layer, and then input them into the pulse index storage buffer Spike_Index_FIFO for subsequent calculation drive excitation. If a pulse recurrent neural network needs to be deployed, the pulse loop buffer Recurrent_Spike_Buffer needs to be enabled, and the pulse data and coded data in the buffer need to be sorted and processed together, and then input into the pulse index storage buffer. If a bidirectional pulse recurrent neural network is deployed, the corresponding data of the input coded data needs to be taken out from the head end and the tail end respectively, and these data are sorted together with the pulse data in the pulse loop memory to obtain the drive excitation of the forward cycle calculation and the drive excitation of the directional cycle calculation, so as to adapt to the deployment and inference acceleration of a variety of different pulse neural network models. For the hardware structure involved in the data flow after the deployment of pulse neural networks with different network topologies, please refer to . Figure 6 .

[0085] It can be found that the main difference between the deployment of a fully connected network and a pulse recurrent neural network is whether a pulse circular buffer is required to circulate the data flow; and the main difference between the deployment of a pulse recurrent neural network and a bidirectional pulse recurrent neural network is whether the reverse flow data needs to be processed.

[0086] The computing system for pulse recurrent neural networks provided by the embodiment of the present application provides a deployment platform for various forms of pulse recurrent neural networks, supports data processing and calculation of different forms of pulse recurrent neural networks; provides a basis for accelerated parallel computing by designing a dual-core neuron group, and optimizes the computing process by improving the LIF neuron; selects a unidirectional or bidirectional computing strategy based on the control module; this process supports parallel computing of complex networks, meets the processing requirements of bidirectional networks, and can also process dual samples of a single task and dual tasks at the same time; in addition, for network architectures that require buffering functions to manage network parameters and parallel operations, this requirement can also be met here through a storage module, which greatly improves the compatibility and adaptability of the deployment platform and can meet the deployment requirements of actual scenarios. Therefore, the embodiment of the present application can solve the technical problem that the existing technology does not support the deployment of multi-layer complex networks and bidirectional networks, nor does it support the simultaneous processing of dual tasks and dual samples, resulting in limited actual deployment operations and low efficiency.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name: Read-Only Memory, English abbreviation: ROM), random access memory (full name: Random Access Memory, English abbreviation: RAM), disk or optical disk, and other media that can store program code.

[0091] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A computing system for pulse recurrent neural networks, characterized in that include: Preprocessing module, core computing module, storage module and control module; The preprocessing module is used to encode different types of target data into 8-bit target pulse data, wherein the target data includes image data, sequence data and text data; The core computing module is used to accelerate parallel computing of the target pulse data through the deployed dual-core neuron group, and select the number of neurons, neuron types and weights according to the computing task. The dual-core neuron group includes two neuron cores, each of which includes two sub-neuron groups connected in parallel. The neuron types include IF neurons and improved LIF neurons. The acquisition process of the improved LIF neurons is as follows: Based on the LIF neuron calculation principle, the original LIF dynamic equation is discretized into a simplified equation to obtain an improved LIF neuron. The simplified equation is expressed as: ； in, is the membrane potential decay rate constant, for When improving the output membrane potential of LIF neurons, for The membrane potential at for The neuron input data at time ; The storage module is used to store the pulse signals, membrane potential information and weights adapted to neurons generated by the core computing module during the computing process; The control module is used to control the preprocessing module to process data, control the core computing module to select neurons and trigger bidirectional computing, and control the storage module to store data.

2. The computing system for pulse recurrent neural networks according to claim 1, characterized in that The preprocessing module includes: An encoding submodule, configured to encode different target data into initial pulse data using different preset encoding algorithms, wherein the preset encoding algorithms include binary encoding, BSA encoding, and rate encoding; The integration submodule is used to compress the pulse signal and merge it into 8-bit real number data to obtain target pulse data.

3. The computing system for pulse recurrent neural networks according to claim 1, wherein: Also includes: Data transmission module; The data transmission module is used to transmit the target pulse data to the FPGA for data calculation through a preset transmission technology, and the preset transmission technology includes a UART interface.

4. The computing system for pulse recurrent neural networks according to claim 1, wherein: Also includes: Data sorting module; The data sorting module is used to perform splicing and sorting processing on the target pulse data to obtain rearranged pulse data, and transmit the rearranged pulse data to the core calculation module for data calculation. The rearranged pulse data includes a pulse index.

5. The computing system for pulse recurrent neural networks according to claim 4, characterized in that The core computing module is specifically used for: Reading the rearranged pulse data from a buffer; Selecting the number and type of neurons according to the computational task, and selecting weights corresponding to the neurons based on the pulse index; The target pulse data is accelerated and parallelized by the deployed dual-core neuron group to obtain a data calculation result, which includes the result pulse data and the result membrane potential.

Images