Storage and calculation integrated neural mimicry hardware-oriented pulse neural network compiling system and method

Abstract

The invention provides a storage and calculation integrated neural mimicry hardware-oriented spiking neural network compiling system and method, and the system comprises a preprocessor part which employs a heuristic algorithm to convert a conventional artificial neural network in an input model into a spiking neural network, and carries out the compiling of an ANN-SNN hybrid network into a computational graph and a neural topology network; and the compiler part is used for compiling a computational graph of the ANN-SNN hybrid network to generate a binary file which can be executed by the storage and calculation integrated neural mimicry hardware, and the compiling stage comprises three universal compiling stages, namely a neuron division stage, a core mapping stage and an instruction generation stage. According to the invention, the problem of storage and calculation integrated neural mimicry hardware acceleration of the ANN-SNN hybrid network is solved, and it is ensured that the ANN-SNN hybrid network can be compiled and submitted to the storage and calculation integrated neural mimicry hardware for acceleration.

Description

Technical Field

[0001] The present invention relates to the fields of compilation framework, hybrid network, neural network, machine learning and PIM accelerator, and in particular, to a pulse neural network compilation system for storage-computation integrated neuromorphic hardware. Background Art

[0002] In recent years, artificial neural networks (ANNs) have developed rapidly and have performed well in a variety of task scenarios. Deep neural networks are currently widely used in various applications, such as medical image analysis, image classification, and natural language processing. However, the high energy consumption of traditional artificial neural networks has become a problem that cannot be ignored. To address this problem, an effective solution is to use spiking neural networks (SNNs) as a substitute. Spiking neural networks imitate the communication and computational methods of human brain neurons, using synapses to interconnect neurons and spikes to transmit information. In spiking neural networks, only addition operations are involved, including adding spikes from synapses to the membrane potential and emitting spikes when the membrane potential exceeds the threshold. Therefore, the energy consumption advantage of spiking neural networks over traditional artificial neural networks is reflected in two aspects: (1) The expensive multiplication operations that frequently occur in traditional artificial neural networks are eliminated. (2) The accumulation of membrane potential is event-driven by spikes, which means that no calculation occurs if no spikes are received.

[0003] However, it is not advisable to only use spiking neural networks and abandon traditional artificial neural networks. Simple vector operations are more efficient in traditional artificial neural network mode, and traditional artificial neural networks can reuse data to reduce memory access. Therefore, using spiking neural networks instead of sub-networks that use a large number of multiplications in traditional artificial neural networks, that is, ANN-SNN hybrid networks, can take advantage of the advantages of both at the same time and is a very promising technology.

[0004] In addition, traditional accelerators, such as central processing units (CPUs) and graphics processing units (GPUs), are also energy-unfriendly and generate a lot of unnecessary energy consumption compared to neuromorphic processors. The main reason is that they do not adopt an event-driven computing model and need to frequently transfer large amounts of data between hardware and memory. Process-in-Memory (PIM) is a technology that integrates the processor and random access memory to perform calculations inside the chip with very little data transmission cost. As a technology that breaks the memory access latency problem inherent in the von Neumann architecture, PIM has begun to be used as acceleration hardware for spiking neural networks and traditional artificial neural networks.

[0005] There are many existing works that use NVM crossbar-based PIM as acceleration hardware for spiking neural networks and traditional artificial neural networks. However, there is little research on the acceleration of ANN-SNN hybrid networks in existing works, and many problems have not been solved. First, to handle ANN-SNN hybrid networks within the compilation framework, a unified description method for hybrid networks is required. Second, the compilation and acceleration of spiking neural networks face some problems that must be considered. If they are not solved, it will bring additional overhead, and there is a lack of a general spiking neural network compilation stage as a guide. Summary of the invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a pulse neural network compilation system and method for storage and computing integrated neuromorphic hardware.

[0007] A pulse neural network compilation method for storage-computation integrated neuromorphic hardware provided by the present invention includes:

[0008] Step S1: converting the artificial neural network into a spiking neural network using a heuristic algorithm to generate a computational graph of the ANN-SNN hybrid network;

[0009] Step S2: Use the compiler to compile the computational graph of the generated ANN-SNN hybrid network to generate a binary file executed by the integrated storage and computing neuromorphic hardware.

[0010] Preferably, step S1 comprises: converting the artificial neural network into a spiking neural network using a heuristic algorithm based on synaptic density selection; and describing the artificial neural network and the spiking neural network in the form of a computational graph.

[0011] Preferably, in the compiler, the operation data of the artificial neural network nodes and the pulse neural network calculation graph are recorded through the HyCOMP_node class, the SNN_info class and the HyCOMP_neuron class;

[0012] Wherein, the HyCOMP_node class stores the data structure of the artificial neural network node and the spiking neural network computation graph node metadata;

[0013] The SNN_info class stores the data structure of the spiking neural network metadata;

[0014] The HyCOMP_neuron class is a data structure that stores information about the neuron topology network in the spiking neural network.

[0015] Preferably, the HyCOMP_node class, the SNN_info class and the HyCOMP_neuron class include sampling interfaces, and the compiler completes data sampling by calling the interfaces.

[0016] Preferably, step S2 comprises:

[0017] Step S2.1: Divide all the strongly connected components in the directed graph of the neuron topology network corresponding to the spiking neural network. When the number of neurons contained in the strongly connected component is less than or equal to the number of neurons that a core can accommodate, there is no need to divide the neurons in the strongly connected component; when the number of neurons contained in the strongly connected component is greater than the number of neurons that a core can accommodate, the neurons in the strongly connected component are divided;

[0018] Step S2.2: After the neuron division is completed, the mapping relationship between neurons and cores is recorded in the core_neuron_map variable;

[0019] Step S2.3: obtaining the topology between the cores based on the mapping relationship between neurons and cores, and representing it using a directed graph; optimizing the topology between the cores represented by the directed graph to obtain the topological structure of the cores on the chip;

[0020] Step S2.4: The artificial neural network nodes and the pulse neural network nodes in the calculation graph based on the ANN-SNN hybrid network use a unified instruction set to generate calculation instructions and control instructions;

[0021] Step S2.5: Record the mapping relationship between neurons and cores and the location of the pulse neural network acceleration core on the chip in the form of a binary file.

[0022] Preferably, the step S2.1 of dividing neurons in the strongly connected component includes:

[0023] When dividing neurons in strongly connected components, an improved depth-first traversal algorithm is used to specify the traversal priority of neurons, from high to low: successor neurons of traversed subgraphs, predecessor neurons of traversed subgraphs, and neurons that are not related to the currently traversed subgraph.

[0024] Preferably, the improved depth-first traversal algorithm includes: when traversing the successors of the current neuron, all successors will be prioritized first, and their priorities will be determined according to the relationship between the successor neurons and the traversed subgraph, from high to low, namely, the successor neurons of the traversed subgraph, the predecessor neurons of the traversed subgraph, and the neurons that are not related to the currently traversed subgraph.

[0025] Preferably, the step S2.3 comprises:

[0026] The weight on the edge indicates how many pairs of neurons between the two cores need to communicate across the cores, y = ∑distance ij *weight ij represents the sum of weighted distances of all cores; the simulated annealing algorithm is used to find the minimum value of y in all solution spaces, and small perturbations are applied to the existing solutions according to the temperature and decay rate until the stop temperature is reached, and the topological structure of the cores on the chip is recorded; where distance ij represents the number of NoC edges that need to be crossed for communication between core i and core j, weight ij Represents the number of neuron pairs that need to communicate across cores between core i and core j.

[0027] Preferably, the calculation instructions in step S2.4 include: operations such as convolution, pooling, matrix dot multiplication, activation function, etc. in artificial neural networks and LIF operations in spiking neural networks;

[0028] The control instructions in step S2.4 include load, store, lmi, lml and instructions for calculation between registers.

[0029] A pulse neural network compilation system for storage-computation integrated neuromorphic hardware provided by the present invention includes:

[0030] Module M1: Use heuristic algorithms to convert artificial neural networks into spiking neural networks and generate computational graphs of ANN-SNN hybrid networks;

[0031] Module M2: Use the compiler to compile the computational graph of the generated ANN-SNN hybrid network to generate a binary file executed by the integrated storage and computing neuromorphic hardware.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) By proposing a unified description method for traditional artificial neural networks and spiking neural networks, the problem that the ANN-SNN hybrid network cannot be uniformly described and compiled is solved, ensuring the correctness of the system.

[0034] (2) By proposing a neuron-divided compilation phase, the problem of excessive cross-core communication after compilation is solved, allowing the system to reduce latency and energy consumption.

[0035] (3) By proposing the core mapping compilation stage, the problem of unnecessary cross-core communication overhead amplification after compilation is solved, allowing the system to reduce latency and energy consumption.

[0036] (4) By proposing the instruction generation compilation stage, the problem of the inability to run correctly due to the dependencies between the computational graph nodes and neurons in the compiled instruction design is solved, thus ensuring the correctness of the system.

[0037] (5) By proposing a heuristic algorithm based on synaptic density selection, the performance degradation problem caused by converting traditional artificial neural network nodes with sparse synapses is solved, allowing the system to reduce latency and energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 This is a compilation framework diagram of the spiking neural network compilation system for storage-computing integrated neuromorphic hardware.

[0040] Figure 2 Schematic diagram of the compilation phase division for the neurons in the compiler part.

[0041] Figure 3 Schematic diagram of the compilation phase for the core mapping of the compiler part.

[0042] Figure 4 Screenshot of the programming part for finding the optimal solution.

[0043] Figure 5 Generates compilation phase diagrams for instructions in the compiler section.

[0044] Figure 6 This is the core member and property graph of the compiler part. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0046] Example 1

[0047] According to the pulse neural network compilation system for storage and computing integrated neuromorphic hardware provided by the present invention, Figures 1 to 6 As shown, including:

[0048] Preprocessor part: Analyze the input traditional artificial neural network, use heuristic algorithm to convert the traditional artificial neural network in the input model into a spiking neural network, generate the computational graph of the ANN-SNN hybrid network, and encode the neurons and synapses in the computational graph.

[0049] Compiler part: In-depth analysis of the network structure, combined with the characteristics of the storage and computing hardware for targeted optimization, compile the pre-processed ANN-SNN hybrid network calculation graph, and generate a binary file that can be executed by the storage and computing neural mimicry hardware. The general compilation stages include: neuron division stage, core mapping stage, and instruction generation stage.

[0050] Specifically, the preprocessor part includes: using a heuristic algorithm based on synaptic density selection to convert the traditional artificial neural network into a pulse neural network; using a unified description method to describe the traditional artificial neural network and the pulse neural network in the form of a computational graph.

[0051] Specifically, the compiler part includes: a data structure HyCOMP_node class that stores metadata of traditional artificial neural network nodes and pulse neural network calculation graph nodes, a data structure SNN_info class that stores metadata of pulse neural network, and a data structure HyCOMP_neuron class that stores information of neuron topology network in pulse neural network.

[0052] Specifically, the compiler part includes three compilation stages, including a neuron partitioning stage, a core mapping stage and an instruction generation stage.

[0053] Specifically, the compiler part records the running data of the traditional artificial neural network nodes and the spiking neural network calculation graph in the compiler through the HyCOMP_node class, SNN_info class and HyCOMP_neuron class;

[0054] The HyCOMP_node class includes:

[0055] bitwidth, which records the bit width of the computational graph node;

[0056] consumer_index, records the index of the successor node of the computational graph node;

[0057] consumer_num, records the number of successor nodes of the computational graph node;

[0058] index, which records the index of the computational graph node;

[0059] input_dim, records the input dimension of the computational graph node;

[0060] input_dim_num, records the number of input dimensions of the computational graph node;

[0061] name, which records the name of the computational graph node;

[0062] Operation records the operation type of the computational graph node;

[0063] output_dim, records the output dimension of the computational graph node;

[0064] output_dim_num, records the number of output dimensions of the computational graph node;

[0065] provider_index, which records the index of the predecessor node of the computational graph node;

[0066] provider_num, records the number of predecessor nodes of the computational graph node;

[0067] consumer_weight, records the weight of the successor node edge of the computational graph node;

[0068] kernel_h, records the convolution kernel height of the computational graph node;

[0069] kernel_w, records the convolution kernel width of the computational graph node;

[0070] stride_h, records the convolution step height of the computational graph node;

[0071] stride_w, records the convolution step width of the computational graph node;

[0072] pad_h0, records the height above the convolution padding of the computational graph node;

[0073] pad_h1, records the height of the convolution padding below the computational graph node;

[0074] pad_w0, records the left width of the convolution padding of the computational graph node;

[0075] pad_w1, records the right width of the convolution padding of the computational graph node;

[0076] input_channel, records the number of input channels of the computational graph node;

[0077] output_channel, records the number of output channels of the computational graph node;

[0078] The HyCOMP_neuron class includes:

[0079] index, records the index of the neuron;

[0080] consumer_num, records the number of successor neurons of this neuron;

[0081] consumer_index_list, records the index of the successor neuron of this neuron;

[0082] consumer_weight_list, which records the weights between this neuron and the subsequent neurons;

[0083] The SNN_info class includes:

[0084] neuron_list, records the node information in the neuron topology network corresponding to the spiking neural network node;

[0085] SNN_graph, records the topological relationship in the neuron topological network corresponding to the spiking neural network node;

[0086] SNN_provider_graph, records the reverse topological relationship of the neuron topological network corresponding to the spiking neural network node;

[0087] disjointed_subgraph_list, records the strongly connected component information of the neuron topology network corresponding to the spiking neural network node;

[0088] core_neuron_map, records the mapping relationship between neurons and cores in the neuron topology network corresponding to the spike neural network node;

[0089] The three classes provide sampling interfaces, and the compiler part can complete data collection by calling the interfaces.

[0090] Specifically, when the compiler compiles traditional artificial neural networks and spiking neural networks, it processes the computational graph nodes according to the order specified in the compilation stage, reducing the latency, energy consumption and other overheads of storage-computing integrated neuromorphic hardware acceleration, including

[0091] Neuron partitioning stage: Divide all strongly connected components in the directed graph of the neuron topology network corresponding to the spiking neural network. When the number of neurons contained in the strongly connected component is less than the number of neurons that a core can accommodate, it means that the cross-core communication of neurons in the strongly connected component can be eliminated. When the number of neurons contained in the strongly connected component is greater than the number of neurons that a core can accommodate, the strongly connected component needs to be further divided. When dividing the strongly connected components, an improved depth-first traversal algorithm is used. When traversing the successors of the current neuron, the algorithm will first prioritize all successors and determine their priorities based on the relationship between the successor neurons and the traversed subgraphs. From high to low, they are the successor neurons of the traversed subgraphs, the predecessor neurons of the traversed subgraphs, and the neurons that are not related to the currently traversed subgraphs. This method achieves the purpose of mapping related neurons to the same core as much as possible, thereby reducing unnecessary cross-core communication. After the neuron partitioning is completed, the mapping relationship between neurons and cores is recorded in the core_neuron_map variable.

[0092] Core mapping stage: The topology between cores is represented by a directed graph. The weights on the edges represent how many pairs of neurons between the two cores need to communicate across the cores. y = ∑distance ij *weight ij Represents the sum of weighted distances of all cores. Use the simulated annealing algorithm to find the minimum value of y in all solution spaces, apply small perturbations to the existing solutions according to the temperature and decay rate until the stop temperature is reached, and record the topological structure of the cores on the chip. ij represents the number of NoC edges that need to be crossed for communication between core i and core j, weight ij Indicates the number of neuron pairs that need to communicate across cores between core i and core j.

[0093] Instruction generation stage: A unified instruction set is used for traditional artificial neural network nodes and pulse neural network nodes in the calculation graph to generate calculation instructions and control instructions. Specifically, memory is allocated to the calculation graph node according to its batch size, number of channels, and tensor size to save the output, and the address of the memory allocated by the system for its predecessor node is obtained to obtain the input. This stage is based on the calculation instructions corresponding to each node in the series settlement graph as the main framework, and multiple control instructions are bound to each calculation instruction to complete the data transmission. The calculation instructions include operations such as convolution, pooling, matrix dot multiplication, activation function in traditional artificial neural networks and LIF operations in pulse neural networks. Control instructions include load (write data in global memory to local memory), store (write data in local memory to global memory), lmi (write constants to local memory), lml (write data in local memory to local memory, similar to memcpy), and instructions for calculation between registers. For neural networks, the topological logic of neurons is difficult to describe in instructions, so it will be recorded in an additional binary file. Written together with the neural network topology is the mapping relationship between neurons and cores, as well as the location of these pulse neural network acceleration cores on the chip.

[0094] The present invention also provides a pulse neural network compilation system for storage-computing integrated neuromorphic hardware. The pulse neural network compilation system for storage-computing integrated neuromorphic hardware can be implemented by executing the process steps of the pulse neural network compilation method for storage-computing integrated neuromorphic hardware, that is, those skilled in the art can understand the pulse neural network compilation method for storage-computing integrated neuromorphic hardware as a preferred implementation of the pulse neural network compilation system for storage-computing integrated neuromorphic hardware.

[0095] Example 2

[0096] Embodiment 2 is a preferred embodiment of Embodiment 1

[0097] like Figure 1 , which is a compilation framework diagram of the spiking neural network compilation system for storage-computation integrated neuromorphic hardware proposed in the proposal of the present invention. The design of the compilation framework mainly involves two parts, the preprocessing part and the compiler part. The preprocessing part describes that the compilation system converts some sub-networks in the input traditional artificial neural network into a spiking neural network, and uniformly describes the traditional artificial neural network and the spiking neural network in the form of a computational graph. The compiler part describes three compilation stages.

[0098] The design of the preprocessing part is described as follows: Before compiling, the compilation system will preprocess the model input by the user to represent it with a unified description of the ANN-SNN hybrid network that is compatible with the computational graph and the neuron-synapse view. The compilation system's description of the pulse neural network is divided into two parts, the SNN node and the neuron topology network. Each SNN node points to a neuron topology network. The SNN node uses the same description as the ANN node so that it can be embedded in the computational graph for representation and processing. The SNN node must not only record basic information such as batch size and number of channels, but also the metadata of the neuron network it represents. The neuron topology network records the topological relationship and weights between neurons in the form of a directed graph.

[0099] The design of the compiler is described as follows: In-depth analysis of the network structure, targeted optimization combined with the characteristics of the storage and computing hardware, compile the preprocessed ANN-SNN hybrid network calculation graph, and generate a binary file that can be executed by the storage and computing integrated neuromorphic hardware. The compilation of the spiking neural network includes three stages: neuron partitioning, core mapping, and instruction generation. The purpose of the neuron partitioning stage is to divide the neurons in an SNN node into multiple subgraphs according to the number of crossbar dendrites to avoid the situation where a spiking neural network contains too many neurons and cannot be assigned to the same core. The core mapping stage is a continuation of the work of the previous stage. In this stage, the mapping relationship between the divided neuron subgraphs and the core will be determined. The instruction generation stage generates specific instructions for each of the traditional artificial neural network and the spiking neural network, and generates other instructions to form a control flow, while recording the topological relationship of the neurons in the spiking neural network and the results of the core mapping.

[0100] like Figure 2 , which is a schematic diagram of the neuron partitioning compilation phase of the compiler part proposed in the proposal of the present invention. The spike output by the predecessor neuron is loaded onto the row of the crossbar by the control unit, and the successor neuron obtains input from the column of the crossbar. Each neuron needs to bind to a crossbar column to obtain input, but the crossbar resources on the core are limited, which means that all neurons in an SNN node cannot be allocated to the same core. The first step in neuron partitioning is to find all the strongly connected components in the directed graph. Ideally, if all neurons in each strongly connected component can be located on the same core, cross-core communication between the predecessor neuron and its successor can be completely avoided to reduce energy consumption and latency. However, in practice, the number of strongly connected component nodes in a directed graph is often large, greater than the number of crossbar columns, and the machine has to continue to be partitioned. Figure 2The coloring of neurons shows the method and process of neuron division. The system chooses to start coloring from neurons with in-degree 0. The neurons with coloring priority from high to low are: successor neurons of the colored subgraph, predecessor neurons of the colored subgraph, and neurons that are not related to the current subgraph. It is worth noting that the compilation system has improved the depth-first traversal. When there are multiple successor neurons, we hope that the next colored neuron is related to the current subgraph, but the depth-first traversal will not record information and may make bad choices, resulting in unnecessary cross-core communication, such as Figure 3 (b) is the ideal mapping situation. Therefore, if Figure 3 As shown by the red edge of the simplified mapping in (a), increasing the priority of the predecessor neurons of the colored subgraph can reduce this situation.

[0101] like Figure 3 As shown in FIG. 1 , it is a schematic diagram of the core mapping compilation phase of the compiler part proposed in the present invention. The goal of core mapping is also to reduce the overhead of cross-core communication. Similar to the neuron topology network, the topology between cores can also be represented by a directed graph, and the weights on the edges represent how many pairs of neurons between the two cores need cross-core communication. ij *weight ij To represent the sum of weighted distances of all cores, the goal is to minimize y. The compiler system searches for the global optimal solution in the solution space. Figure 4 The algorithm shown in the figure shows the process of finding the optimal solution. T, α and T' represent temperature, decay rate and stop temperature respectively. According to the temperature and decay rate, the positions of the two cores are randomly exchanged based on the existing solution to try to get a smaller result. If so, the existing solution is updated until the stop temperature is reached and the topology of the core on the chip is recorded.

[0102] like Figure 5As shown, it is a schematic diagram of the instruction generation compilation stage of the compiler part proposed in the proposal of the present invention. The description of the pulse neural network by the compilation system is divided into two parts: SNN nodes and neuron topology networks. For SNN nodes and ANN nodes, the compilation system uses a unified instruction set. Computational instructions include operation nodes such as convolution, pooling, matrix dot multiplication, activation function, etc. in traditional artificial neural networks and LIF operations in pulse neural networks. These instructions, like the nodes in the calculation graph, have dependencies on the order of execution. When two instructions with dependencies are assigned to different cores for acceleration, the consumer needs to wait for the execution result of the producer. In addition, there are many intra-core communications and cross-core communications in the acceleration process of the ANN-SNN hybrid network, so control instructions for loading and storing data are also required, including the interaction between the local memory of the core and the on-chip global memory, the interaction between the local memories of the two cores, and the writing of data on the crossbar by the pulse neural network acceleration core. For neural networks, the topological logic of neurons is difficult to describe in instructions, so it will be recorded in additional binary files. Written along with the neural network topology is the mapping relationship between neurons and cores, as well as the location of these spiking neural network acceleration cores on the chip.

[0103] like Figure 6, which are the core members and attribute graphs of the compiler part proposed by the proposal of the present invention. The operation data of the traditional artificial neural network nodes and the pulse neural network calculation graph in the compiler are recorded through the HyCOMP_node class, the SNN_info class and the HyCOMP_neuron class. The HyCOMP_node class includes: bitwidth records the bit width of the calculation graph node, consumer_index records the successor node index of the calculation graph node, consumer_num records the number of successor nodes of the calculation graph node, index records the index of the calculation graph node, input_dim records the input dimension of the calculation graph node, input_dim_num records the number of input dimensions of the calculation graph node, name records the name of the calculation graph node, operation records the operation type of the calculation graph node, output_dim records the output dimension of the calculation graph node, output_dim_num records the number of output dimensions of the calculation graph node, provi der_index records the predecessor node index of the computational graph node, provider_num records the number of predecessor nodes of the computational graph node, consumer_weight records the weight of the successor node edge of the computational graph node, kernel_h records the convolution kernel height of the computational graph node, kernel_w records the convolution kernel width of the computational graph node, stride_h records the convolution step height of the computational graph node, stride_w records the convolution step width of the computational graph node, pad_h0 records the convolution padding top height of the computational graph node, pad_h1 records the convolution padding bottom height of the computational graph node, pad_w0 records the convolution padding left width of the computational graph node, pad_w1 records the convolution padding right width of the computational graph node, input_channel records the number of input channels of the computational graph node, and output_channel records the number of output channels of the computational graph node. The HyCOMP_neuron class includes: index records the index of the neuron, consumer_num records the number of successor neurons of the neuron, consumer_index_list records the index of the successor neurons of the neuron, and consumer_weight_list records the weight between the neuron and the successor neurons.The SNN_info class includes: neuron_list records the node information in the neuron topology network corresponding to the spiking neural network node, SNN_graph records the topological relationship in the neuron topology network corresponding to the spiking neural network node, SNN_provider_graph records the reverse topological relationship of the neuron topology network corresponding to the spiking neural network node, disjointed_subgraph_list records the strongly connected component information of the neuron topology network corresponding to the spiking neural network node, and core_neuron_map records the mapping relationship between neurons and cores in the neuron topology network corresponding to the spiking neural network node. The three classes provide sampling interfaces, and the compiler part can complete data collection by calling the interface.

[0104] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0105] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A spiking neural network compilation method for storage-computation integrated neuromorphic hardware, characterized in that: include: Step S1: converting the artificial neural network into a spiking neural network using a heuristic algorithm to generate a computational graph of the ANN-SNN hybrid network; Step S2: Use the compiler to compile the computational graph of the generated ANN-SNN hybrid network to generate a binary file executed by the integrated storage and computing neuromorphic hardware.

2. The spiking neural network compilation method for storage-computation integrated neuromorphic hardware according to claim 1 is characterized in that: The step S1 includes: using a heuristic algorithm based on synaptic density selection to convert the artificial neural network into a spiking neural network; and describing the artificial neural network and the spiking neural network in the form of a computational graph.

3. The spiking neural network compilation method for storage-computation integrated neuromorphic hardware according to claim 1 is characterized in that: In the compiler, the operation data of the artificial neural network nodes and the pulse neural network calculation graph are recorded through the HyCOMP_node class, the SNN_info class and the HyCOMP_neuron class; Wherein, the HyCOMP_node class stores the data structure of the artificial neural network node and the spiking neural network computation graph node metadata; The SNN_info class stores the data structure of the spiking neural network metadata; The HyCOMP_neuron class is a data structure that stores information about the neuron topology network in the spiking neural network.

4. The spiking neural network compilation method for storage-computation-in-one neuromorphic hardware according to claim 1 is characterized in that: The HyCOMP_node class, the SNN_info class and the HyCOMP_neuron class include sampling interfaces, and the compiler completes data sampling by calling the interfaces.

5. The spiking neural network compilation method for storage-computation integrated neuromorphic hardware according to claim 1 is characterized in that: The step S2 comprises: Step S2.1: Divide all the strongly connected components in the directed graph of the neuron topology network corresponding to the spiking neural network. When the number of neurons contained in the strongly connected component is less than or equal to the number of neurons that a core can accommodate, there is no need to divide the neurons in the strongly connected component; when the number of neurons contained in the strongly connected component is greater than the number of neurons that a core can accommodate, the neurons in the strongly connected component are divided; Step S2.2: After the neuron division is completed, the mapping relationship between neurons and cores is recorded in the core_neuron_map variable; Step S2.3: obtaining the topology between the cores based on the mapping relationship between neurons and cores, and representing it using a directed graph; optimizing the topology between the cores represented by the directed graph to obtain the topological structure of the cores on the chip; Step S2.4: The artificial neural network nodes and the pulse neural network nodes in the calculation graph based on the ANN-SNN hybrid network use a unified instruction set to generate calculation instructions and control instructions; Step S2.5: Record the mapping relationship between neurons and cores and the location of the pulse neural network acceleration core on the chip in the form of a binary file.

6. The spiking neural network compilation method for storage-computation-in-one neuromorphic hardware according to claim 5 is characterized in that: The step S2.1 of dividing neurons in the strongly connected component includes: When dividing neurons in strongly connected components, an improved depth-first traversal algorithm is used to specify the traversal priority of neurons, from high to low: successor neurons of traversed subgraphs, predecessor neurons of traversed subgraphs, and neurons that are not related to the currently traversed subgraph.

7. The spiking neural network compilation method for storage-computation-in-one neuromorphic hardware according to claim 6 is characterized in that: The improved depth-first traversal algorithm includes: when traversing the successors of the current neuron, all successors will be prioritized first, and their priorities will be determined according to the relationship between the successor neurons and the traversed subgraphs, from high to low, the successor neurons of the traversed subgraphs, the predecessor neurons of the traversed subgraphs, and the neurons that are not related to the currently traversed subgraphs.

8. The spiking neural network compilation method for storage-computation-in-one neuromorphic hardware according to claim 5 is characterized in that: The step S2.3 comprises: The weight on the edge indicates how many pairs of neurons between the two cores need to communicate across the cores, y = ∑distance ij *weight ij represents the sum of weighted distances of all cores; the simulated annealing algorithm is used to find the minimum value of y in all solution spaces, and small perturbations are applied to the existing solutions according to the temperature and decay rate until the stop temperature is reached, and the topological structure of the cores on the chip is recorded; where distance ij represents the number of NoC edges that need to be crossed for communication between core i and core j, weight ij Represents the number of neuron pairs that need to communicate across cores between core i and core j.

9. The spiking neural network compilation method for storage-computation-in-one neuromorphic hardware according to claim 5 is characterized in that: The calculation instructions in step S2.4 include: operations such as convolution, pooling, matrix dot multiplication, activation function, etc. in artificial neural networks and LIF operations in spiking neural networks; The control instructions in step S2.4 include load, store, lmi, lml and instructions for calculation between registers.

10. A spiking neural network compilation system for storage-computation integrated neuromorphic hardware, characterized in that: include: Module M1: Use heuristic algorithms to convert artificial neural networks into spiking neural networks and generate computational graphs of ANN-SNN hybrid networks; Module M2: Use the compiler to compile the computational graph of the generated ANN-SNN hybrid network to generate a binary file executed by the integrated storage and computing neuromorphic hardware.

Images