A pulse neural network compiling method and system for a brain-like chip

Abstract

The application discloses a kind of brain-like chip-oriented pulse neural network compilation method, it is expanded pulse neural network Relay operator in TVM, access and bind to hardware operator library, automatically realize from pulse neural network model to brain-like chip adaptation intermediate representation conversion and optimization, in combination with the optimization method of operator fusion optimization for pulse neural network model, data layout conversion for brain-like chip, make full use of the high-performance implementation of hardware operator library;Finally, based on the infrastructure of TVM, LLVM compiler generates deployment file package, efficiently completes the compilation and deployment of pulse neural network model on brain-like chip.The application can solve the technical problems of high learning cost, low development efficiency existing in the existing DSL-based neural network compilation method and the neural network compilation method based on hardware intrinsic function or operator API.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and more specifically, relates to a method and system for compiling a spiking neural network (SNN) for neuromorphic chips. Background Technology

[0002] Spiking Neural Networks (SNNs) possess natural advantages in processing time-series data and optimizing energy efficiency by mimicking the working mechanisms of biological nerves. They can be used to implement complex artificial intelligence applications such as image classification and speech recognition. The basic elements of an SNN include spikes, neurons, and synapses, which together constitute the core mechanism for simulating biological nervous systems. SNN operators are commonly used to describe the basic functional units that describe the computational behavior of neurons and synapses, such as fully connected operators, convolutional connection operators, and leaky-integrate-fire (LIF) neuron operators.

[0003] Neuromorphic chips, based on Sense Neural Networks (SNNs), simulate the information processing methods of the human brain through hardware circuits, enabling behaviors such as perception, thinking, and decision-making. They feature low power consumption, in-memory computing, and high parallelism, and show broad application prospects in multiple fields.

[0004] Because neuromorphic chips are specifically designed for SNNs (Small Neural Networks), they are typically highly customized, making the deployment of SNNs on neuromorphic chips a significant challenge. Existing methods for deploying neural network models to neuromorphic chips (i.e., neural network compilation methods) mainly include the following: The first is a neural network compilation method based on hardware intrinsic functions (compiler-built-in functions) or operator application programming interfaces (APIs), which manually deploys the model by calling intrinsic functions or operator APIs. The second is a neural network compilation method based on a domain-specific language (DSL). DSLs are programming languages ​​specifically designed for a particular domain or hardware. DSLs provide high-level model abstraction interfaces, and by describing the high-level logic of the model, the DSL compiler automatically completes the underlying hardware optimization and deployment. The third method is a neural network compilation method based on intermediate representation (IR). This method converts the neural network model into IR and performs a series of hardware-independent and hardware-dependent optimizations based on the IR, ultimately generating hardware-executable code.

[0005] However, the aforementioned neural network compilation methods all have some significant drawbacks:

[0006] First, neuromorphic chip hardware design is typically highly customized, with complex details regarding neuron models, spike and weight formats, and synaptic computation rules. Existing neural network compilation methods based on hardware intrinsic functions or operator APIs require developers to have a deep understanding of the underlying hardware architecture of neuromorphic chips, resulting in high learning costs and development barriers, low development efficiency, high error rates, and a lack of scalability. Existing DSL-based neural network compilation methods also require learning new development languages ​​and programming paradigms, which, while lowering the development barrier, still have high learning costs.

[0007] Second, existing IR-based neural network compilation methods are mainly geared towards Artificial Neural Networks (ANNs) and lack end-to-end compilation support for SNNs and neuromorphic chips. Mainstream neural network compilers such as Tensor Virtual Machines (TVMs) have not yet provided SNN compilation support, thus lacking optimization measures for the characteristics of SNNs and neuromorphic chips.

[0008] Third, designing and developing neural network compilers based on DSL or IR from scratch requires comprehensive customization for the characteristics of SNN and neuromorphic chips. This is challenging, time-consuming, and lacks scalability and reusability. Summary of the Invention

[0009] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a spiking neural network compilation method and system for neuromorphic chips. Its purpose is to solve the technical problems of high learning costs and low development efficiency in existing DSL-based neural network compilation methods and hardware intrinsic function or operator API-based neural network compilation methods; the lack of optimization for the characteristics of SNNs and neuromorphic chips in existing IR-based neural network compilation methods; and the technical problems of existing IR-based and DSL-based neural network compilation methods requiring custom design from scratch for SNNs and neuromorphic chips, resulting in high design difficulty, long development cycles, and insufficient scalability and reusability in developing dedicated neural network compilers.

[0010] To achieve the above objectives, according to one aspect of the present invention, a method for compiling a spiking neural network (SNN) for neuromorphic chips is provided, comprising the following steps:

[0011] (1) Obtain the neuromorphic chip operator library, register the corresponding Relay operator for each operator in the neuromorphic chip operator library based on the TVM infrastructure, and establish an association between the Relay operator and the corresponding operator in the neuromorphic chip operator library to obtain the Relay operator library adapted to the neuromorphic chip.

[0012] (2) Obtain the SNN model described by the high-level language development framework, and perform model transformation processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain the Relay intermediate representation Relay IR which is equivalent to the SNN model.

[0013] (3) The Relay IR obtained in step (2) is subjected to deformation processing oriented towards SNN model and neuromorphic chip characteristics to obtain the deformed Relay IR.

[0014] (4) Use the LLVM backend code generator in the TVM infrastructure to perform sinking processing on the Relay IR after the data layout transformation obtained in step (3) to obtain LLVM intermediate code.

[0015] (5) Use the TVM infrastructure to compile the Relay IR after the data layout transformation obtained in step (3) into model files and weight files, and use the LLVM compiler that supports the backend of neuromorphic chips to compile the LLVM intermediate code obtained in step (4) into operator files. Then, package the model files, weight files and operator files into the final deployment file package as the final SNN compilation result.

[0016] Preferably, step (1) involves obtaining a neuromorphic chip operator library from an existing neuromorphic chip development toolchain. The neuromorphic chip operator library is a set of operators optimized for neuromorphic chips.

[0017] Preferably, step (2) includes the following sub-steps:

[0018] (2-1) Obtain the SNN model described by the high-level language development framework and convert it into a directed acyclic graph (DAG). The nodes in the DAG are the neuron clusters in the SNN model, and the edges in the DAG are the connections between the neuron clusters in the SNN model.

[0019] (2-2) Use topological sorting to traverse all nodes in the DAG obtained in step (2-1) to obtain an unprocessed node N;

[0020] (2-3) Obtain all input edges of node N in DAG obtained in step (2-2), and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set.

[0021] (2-4) Determine whether the number of Relay operator nodes in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulating Relay operator node, use all Relay operator nodes in the Relay operator node set as input to the accumulating Relay operator node, and use the accumulating Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5).

[0022] (2-5) Based on node N obtained in step (2-2), construct a leak-accumulate-excite LIF Relay operator node, and use the summary result obtained in step (2-4) as the input of the LIF Relay operator node. The LIF Relay operator node is the Relay operator node equivalent to node N.

[0023] (2-6) For all remaining unprocessed nodes in the DAG, repeat steps (2-2) to (2-5) until all unprocessed nodes in the DAG have been processed, thus finally obtaining a Relay IR that is equivalent to the SNN model described by the high-level language development framework.

[0024] Preferably, step (2-3) specifically involves the following steps: First, for each input edge of node N in the DAG graph obtained in step (2-2), the synaptic type information of the input edge is obtained; then, based on the synaptic type information of the input edge, the corresponding Relay operator is obtained from the Relay operator library, and the corresponding Relay operator node currNode is constructed based on the Relay operator; subsequently, the Relay operator node prevNode corresponding to the starting point (i.e., the predecessor node) of the input edge in the DAG graph is obtained, and the Relay operator node prevNode is used as the input of the Relay operator node currNode, establishing a dependency relationship between the Relay operator node prevNode and the Relay operator node currNode; then, for all remaining input edges of node N in the DAG graph, the above process is repeated, and all constructed Relay operator nodes are summarized into a Relay operator node set.

[0025] Preferably, step (3) includes the following steps:

[0026] (3-1) Obtain the neuromorphic chip operator library. Based on the fusion operator types in the neuromorphic chip operator library, summarize the SNN operator fusion modes supported by the neuromorphic chip. Based on the SNN operator fusion modes, register the SNN operator fusion modes and corresponding rewrite functions using the TVM infrastructure to obtain the SNN operator fusion mode library.

[0027] (3-2) Using the SNN operator fusion pattern library obtained in step (3-1), perform operator fusion processing on the Relay IR obtained in step (2) to obtain the Relay IR after operator fusion. Examples of the changes in Relay IR before and after operator fusion are shown below. Figure 4 As shown.

[0028] (3-3) Perform data layout transformation on the Relay IR obtained by operator fusion in step (3-2) to obtain the Relay IR after data layout transformation.

[0029] Preferably, step (3-2) specifically involves: first, obtaining the SNN operator fusion pattern library defined in step (3-1), and using the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, using the MergeComposite optimization pass in the TVM infrastructure to perform subgraph matching processing on the Relay IR obtained in step (2) to obtain a Relay subgraph RSG that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; next, obtaining the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and using the rewrite function to rewrite the RSG into an operator-fused Relay subgraph; finally, adding a label to the operator-fused Relay subgraph to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate identification and processing in the subsequent code generation stage, thereby obtaining the operator-fused Relay IR.

[0030] Preferably, step (3-3) specifically involves first obtaining all weight data from the Relay IR obtained after operator fusion in step (3-2), and then converting all weight data into a format compatible with the neuromorphic chip operator library according to the data layout format requirements of the neuromorphic chip operator library, so as to obtain the Relay IR after data layout conversion.

[0031] Preferably, step (4) specifically involves using the LLVM backend code generator in the TVM infrastructure to compile each Relay operator node in the Relay IR after the data layout conversion obtained in step (3) into the corresponding operator call statement in the neuromorphic chip operator library, and combining all the compiled call statements to obtain the LLVM intermediate code.

[0032] According to another aspect of the present invention, a spiking neural network (SNN) compilation system for neuromorphic chips is provided, comprising:

[0033] The first module is used to obtain the neuromorphic chip operator library. Based on the TVM infrastructure, it registers the corresponding Relay operator for each operator in the neuromorphic chip operator library and establishes an association between the Relay operator and the corresponding operator in the neuromorphic chip operator library to obtain a Relay operator library adapted to neuromorphic chips.

[0034] The second module is used to obtain the SNN model described by the high-level language development framework, and to perform model transformation processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in the first module, so as to obtain the Relay intermediate representation Relay IR which is equivalent to the SNN model.

[0035] The third module is used to perform deformation processing on the Relay IR obtained from the second module to be oriented towards the SNN model and the characteristics of neuromorphic chips, so as to obtain the deformed Relay IR.

[0036] The fourth module is used to use the LLVM backend code generator in the TVM infrastructure to perform downsinking processing on the Relay IR obtained from the data layout transformation in the third module, in order to obtain LLVM intermediate code.

[0037] The fifth module is used to compile the Relay IR obtained from the data layout transformation in the third module into model files and weight files using the TVM infrastructure, and to compile the LLVM intermediate code obtained from the fourth module into operator files using the LLVM compiler that supports the neuromorphic chip backend. Finally, the model files, weight files, and operator files are packaged into the final deployment file package as the final SNN compilation result.

[0038] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0039] (1) This invention employs steps (1) and (2), which register the SNN Relay operator in TVM and establish an association with the operator in the neuromorphic chip operator library, thereby connecting the SNN model described by the high-level language development framework to the TVM front end and realizing TVM's compilation support for the SNN model. Therefore, it can solve the technical problems of high learning cost and low development efficiency of existing neural network compilation methods based on DSL and neural network compilation methods based on hardware intrinsic functions or operator APIs.

[0040] (2) Since the present invention adopts step (3), it adds IR transformation processing based on TVM infrastructure for SNN model and neuromorphic chip characteristics, thus solving the technical problem that existing IR-based neural network compilation methods lack optimization for SNN and neuromorphic chip characteristics.

[0041] (3) Since the present invention adopts steps (1) to (5), it achieves the end-to-end compilation process from the SNN model described by the high-level language framework to the underlying machine code by reusing the high-level language framework, the neuromorphic chip operator library, the TVM infrastructure and the LLVM compiler. This solves the technical problems of existing neural network compilation methods based on IR and neural network compilation methods based on DSL, which require custom design from scratch for SNN and neuromorphic chips, and the development of dedicated neural network compilers is difficult, has a long development cycle, and lacks scalability and reusability. Attached Figure Description

[0042] Figure 1 This is a flowchart of the spiking neural network compilation method for neuromorphic chips according to the present invention;

[0043] Figure 2 This is a flowchart of step (2) in the method of the present invention;

[0044] Figure 3 This is a schematic diagram of step (2) in the method of the present invention;

[0045] Figure 4 This is a schematic diagram of the Relay IR before and after operator fusion optimization according to the present invention;

[0046] Figure 5 This is a schematic diagram of the weight data of the present invention before and after data layout transformation and optimization;

[0047] Figure 6 This is a flowchart of the code generation and basic compilation of this invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Spiking neural networks are considered to have higher biological interpretability and are better adapted to low-power neuromorphic chips. Neuromorphic chips theoretically have the characteristics of low power consumption, high robustness, high efficiency parallelism, and self-adaptability. However, the technical solutions have not yet been unified and are usually highly customized. Deploying spiking neural network models on neuromorphic chips has high learning costs and development difficulties, and there is a lack of end-to-end model compilation and deployment tools.

[0050] The basic idea of ​​this invention is to provide a spiking neural network compilation method and system for neuromorphic chips, taking into account the characteristics of spiking neural network models. By extending the spiking neural network relay operator in TVM and connecting and binding it to the hardware operator library, the intermediate representation conversion and optimization from the spiking neural network model to the neuromorphic chip is automatically realized. Combined with optimization methods such as operator fusion optimization for spiking neural network models and data layout conversion for neuromorphic chips, the high performance of the hardware operator library is fully utilized. Finally, based on the TVM infrastructure and LLVM compiler, a deployment package is generated to efficiently complete the compilation and deployment of the spiking neural network model on the neuromorphic chip.

[0051] To make the objectives and features of the present invention more apparent and easier to understand, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] like Figure 1 As shown, this invention provides a method for compiling a spiking neural network (SNN) for neuromorphic chips, comprising the following steps:

[0053] (1) Obtain the neuromorphic chip operator library, register the corresponding Relay operator for each operator in the neuromorphic chip operator library based on the TVM infrastructure, and establish an association between the Relay operator and the corresponding operator in the neuromorphic chip operator library to obtain the Relay operator library adapted to the neuromorphic chip.

[0054] Specifically, this step involves obtaining a neuromorphic chip operator library from an existing neuromorphic chip development toolchain. This neuromorphic chip operator library is a collection of operators optimized for neuromorphic chips, containing multiple operator implementations customized for hardware architectures, and covering commonly used operator implementations in spiking neural networks.

[0055] (2) Obtain the SNN model described by the high-level language development framework, and perform model transformation processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain the Relay Intermediate Representation (Relay IR) which is equivalent to the SNN model.

[0056] like Figure 2 As shown, this step includes the following sub-steps:

[0057] (2-1) Obtain the SNN model described by the high-level language development framework and convert it into a Directed Acyclic Graph (DAG). The nodes in the DAG are the neuron clusters in the SNN model, and the edges in the DAG are the connections between the neuron clusters in the SNN model.

[0058] (2-2) Use topological sorting to traverse all nodes in the DAG obtained in step (2-1) to obtain an unprocessed node N;

[0059] (2-3) Obtain all input edges of node N in DAG obtained in step (2-2), and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set.

[0060] Specifically, this step involves the following steps: First, for each input edge of node N in the DAG graph obtained in step (2-2), the synaptic type information of the input edge is obtained. Then, based on the synaptic type information of the input edge, the corresponding Relay operator is obtained from the Relay operator library, and the corresponding Relay operator node currNode is constructed based on the Relay operator. Subsequently, the Relay operator node prevNode corresponding to the starting point (i.e., the predecessor node) of the input edge in the DAG graph is obtained, and the Relay operator node prevNode is used as the input of the Relay operator node currNode, establishing a dependency relationship between the Relay operator node prevNode and the Relay operator node currNode. Afterward, for all the remaining input edges of node N in the DAG graph, the above process is repeated, and all constructed Relay operator nodes are summarized into a Relay operator node set for subsequent node transformations.

[0061] (2-4) Determine whether the number of Relay operator nodes in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an Accumulation Relay operator node, use all Relay operator nodes in the Relay operator node set as input to the Accumulation Relay operator node, and use the Accumulation Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5).

[0062] (2-5) Based on node N obtained in step (2-2), construct a Leaky-Integrate-Fire (LIF) Relay operator node, and use the summary result obtained in step (2-4) as the input of the LIF Relay operator node. The LIF Relay operator node is the Relay operator node equivalent to node N.

[0063] (2-6) For all remaining unprocessed nodes in the DAG, repeat steps (2-2) to (2-5) until all unprocessed nodes in the DAG have been processed, thus finally obtaining a Relay IR equivalent to the SNN model described by the high-level language development framework. An example of converting an SNN model to a Relay IR is shown below. Figure 3 As shown.

[0064] The advantages of the above steps (1) to (2) are that the dedicated Relay operator in the SNN field is registered based on the TVM infrastructure and associated with the neuromorphic chip operator library, and the SNN described by the high-level language development framework is converted into Relay IR, so that developers do not need to deeply understand the underlying hardware architecture and can directly use the high-level language development framework to develop SNN models, which greatly reduces the learning cost and development threshold.

[0065] (3) The Relay IR obtained in step (2) is transformed to be oriented towards the SNN model and the characteristics of the neuromorphic chip, so as to obtain the transformed Relay IR.

[0066] This step (3) includes the following steps:

[0067] (3-1) Obtain the neuromorphic chip operator library. Based on the fusion operator types in the neuromorphic chip operator library, summarize the SNN operator fusion modes supported by the neuromorphic chip. Based on the SNN operator fusion modes, register the SNN operator fusion modes and corresponding rewrite functions using the TVM infrastructure to obtain the SNN operator fusion mode library.

[0068] Specifically, the neuromorphic chip operator library mentioned in this step contains multiple efficient fusion operators customized for neuromorphic chips. By analyzing the fusion operators in the neuromorphic chip operator library, the SNN operator fusion modes supported by neuromorphic chips are extracted. In this example, the SNN operator fusion modes include: fully connected-LIF mode, convolutional connection-LIF mode, max pooling connection-LIF mode, etc.

[0069] (3-2) Using the SNN operator fusion pattern library obtained in step (3-1), perform operator fusion processing on the Relay IR obtained in step (2) to obtain the Relay IR after operator fusion. Examples of the changes in Relay IR before and after operator fusion are shown below. Figure 4 As shown.

[0070] Specifically, this step involves: first, obtaining the SNN operator fusion pattern library defined in step (3-1), and using the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, using the MergeComposite optimization pass in the TVM infrastructure to perform subgraph matching processing on the Relay IR obtained in step (2) to obtain a Relay subgraph (RSG) that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; next, obtaining the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and using the rewrite function to rewrite the RSG into an operator-fused Relay subgraph; finally, adding a tag to the operator-fused Relay subgraph to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate identification and processing in the subsequent code generation stage, thereby obtaining the operator-fused Relay IR;

[0071] (3-3) Perform data layout transformation on the Relay IR obtained by operator fusion in step (3-2) to obtain the Relay IR after data layout transformation.

[0072] Specifically, this step involves first obtaining all weight data from the Relay IR obtained after operator fusion in step (3-2). Then, based on the data layout format requirements of the neuromorphic chip operator library, all weight data are converted to a format compatible with the neuromorphic chip operator library to obtain the Relay IR after data layout conversion. An example of the changes in weight data layout before and after conversion is shown below. Figure 5 As shown.

[0073] The advantage of this step (3) is that by adding the SNN operator fusion mode supported by the neuromorphic chip to the TVM, the subgraph matching mechanism of the TVM is used to automatically realize the matching and rewriting of the Relay subgraph, and multiple operators are merged into an efficient fusion operator, making full use of the efficient operator implementation of the neuromorphic chip hardware operator library; by adding the data layout conversion optimization process, the weight data format is adjusted to the layout format supported by the neuromorphic chip operator library, avoiding the extra overhead caused by data format mismatch, and solving the problem that the existing neural network compiler lacks optimization for SNN and neuromorphic chip characteristics;

[0074] (4) Use the LLVM backend code generator in the TVM infrastructure to perform a lowering process on the Relay IR after the data layout transformation obtained in step (3) to obtain the LLVM intermediate code.

[0075] Specifically, this step involves using the LLVM backend code generator in the TVM infrastructure to compile each Relay operator node in the Relay IR after the data layout conversion obtained in step (3) into the corresponding operator call statement in the neuromorphic chip operator library, and then combining all the compiled call statements to obtain the LLVM intermediate code.

[0076] (5) Using the TVM infrastructure, the Relay IR obtained in step (3) after data layout transformation is compiled into model files and weight files. Then, using the LLVM compiler that supports the neuromorphic chip backend, the LLVM intermediate code obtained in step (4) is compiled into operator files. The model files, weight files, and operator files are packaged into the final deployment file package as the final SNN compilation result. The processes of steps (4) and (5) are as follows: Figure 6 As shown.

[0077] The advantages of steps (1) to (5) above are that by reusing high-level language frameworks and neuromorphic chip operator libraries, combined with the existing compilation infrastructure of TVM and LLVM, the high cost and high complexity of building a compiler from scratch are effectively avoided; by establishing a fully automated end-to-end compilation process from high-level language frameworks to low-level hardware code generation, the workload of developers in model conversion, optimization and deployment is reduced, and the development cycle is significantly shortened.

[0078] Example

[0079] The neuromorphic chip platform involved in this embodiment is NeuroRVcore, a neuromorphic chip based on the RISC-V instruction set. Specifically, NeuroRVcore extends the neuromorphic computing microarchitecture and instruction set based on the RISC-V microprocessor, supports parallel computing of up to 1024 configurable LIF neurons, and supports synaptic computing at various granularities. It features low power consumption and strong scalability. This neuromorphic chip platform supports model development based on the neuromorphic chip operator library (hereinafter referred to as the "NeuroRVcore operator library") and supports C / C++ and LLVM code compilation.

[0080] This embodiment is used to further illustrate the present invention. The following are the specific implementation steps of Embodiment 2:

[0081] (1) Obtain the NeuroRVcore operator library from the NeuroRVcore development toolchain, register a Relay operator for each operator in the NeuroRVcore operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the neuromorphic chip operator library to obtain a Relay operator library adapted to NeuroRVcore. Specifically, the NeuroRVcore operator library contains multiple high-performance operator implementations for the NeuroRVcore neuromorphic chip. The NeuroRVcore operator library includes, but is not limited to, the following operators: fully connected operator, accumulation operator, LIF operator, and fully connected-LIF fusion operator;

[0082] (2) Obtain the SNN model described by the high-level language development framework BindsNet, and perform model transformation processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain the RelayIR equivalent to the SNN model;

[0083] This step specifically includes the following steps:

[0084] (2-1) Obtain the SNN model described by the high-level language development framework BindsNet and convert it into a DAG.

[0085] Specifically, based on the SNN model source code described by BindsNet, the Python interpreter is used to obtain the Network class object in the BindsNet high-level language development framework. The Network class object stores the information of the user-defined SNN model. Obtaining the Network class object can yield the DAG formed by the SNN model. Specifically, the layers data structure in the Network stores the information of neuron clusters, and the connections data structure in the Network stores the information of the connections between neuron clusters.

[0086] (2-2) Traverse all nodes in the DAG obtained in step (2-1) in topological order to obtain an unprocessed node N;

[0087] (2-3) Obtain all input edges of node N in DAG obtained in step (2-2), and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set.

[0088] Specifically, this step involves the following steps: If the synaptic type of an input edge of node N is fully connected, then a fully connected Relay operator is retrieved from the Relay operator library, and a fully connected Relay operator node FCNode is constructed (for other synaptic types, they are mapped to the corresponding Relay operators in the Relay operator library, which will not be elaborated here); then, the Relay operator node LIFNode corresponding to the starting point (i.e., the predecessor node) of this input edge in the DAG graph is retrieved, and LIFNode is used as the input of FCNode, establishing the dependency relationship between LIFNode and FCNode. Finally, for all remaining input edges of node N in the DAG graph, the above process is repeated, and all constructed Relay operator nodes are summarized into a Relay operator node set for subsequent node transformations.

[0089] (2-4) Determine whether the number of elements in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulating Relay operator node, use all Relay operator nodes in the Relay operator node set as input to the accumulating Relay operator node, and use the accumulating Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5).

[0090] (2-5) Obtain node N obtained in step (2-2), construct a LIF Relay operator node, and use the summary result obtained in step (2-4) as the input of the LIF Relay operator node to obtain a Relay operator node equivalent to node N;

[0091] (2-6) For all remaining unprocessed nodes in the DAG, repeat steps (2-2) to (2-5) until all unprocessed nodes in the DAG have been processed, thus finally obtaining a Relay IR that is equivalent to the SNN model described by the high-level language development framework.

[0092] (3) Perform transformation processing on the Relay IR obtained in step (2) to adapt it to the SNN model and NeuroRVcore characteristics, so as to obtain the transformed Relay IR.

[0093] Specifically, this step includes the following sub-steps:

[0094] (3-1) Obtain the NeuroRVcore operator library, obtain the SNN operator fusion mode supported by the neuromorphic chip according to the fusion operator type in the NeuroRVcore operator library, and register the rewrite function corresponding to the SNN operator fusion mode using the TVM infrastructure to obtain the SNN operator fusion mode library.

[0095] Specifically, the NeuroRVcore operator library mentioned in this step contains several efficient fusion operators customized for NeuroRVcore. By analyzing the fusion operators in the neuromorphic chip operator library, the SNN operator fusion mode supported by the neuromorphic chip is extracted. In this example, the SNN operator fusion mode includes, but is not limited to, the fully connected-LIF mode.

[0096] (3-2) Use the SNN operator fusion mode library obtained in step (3-1) to perform SNN operator fusion processing on the Relay IR obtained in step (2) to obtain the Relay IR after operator fusion.

[0097] Specifically, this step involves: first, obtaining the SNN operator fusion pattern library defined in step (3-1), and using the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, using the MergeComposite optimization pass in the TVM infrastructure to perform subgraph matching processing on the Relay IR obtained in step (2) to obtain a Relay subgraph (RSG) that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; next, obtaining the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and using the rewrite function to rewrite the RSG into an operator-fused Relay subgraph; finally, adding a tag to the operator-fused Relay subgraph to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate identification and processing in the subsequent code generation stage, thereby obtaining the operator-fused Relay IR;

[0098] (3-3) Perform data layout transformation on the Relay IR obtained by operator fusion in step (3-2) to obtain the Relay IR after data layout transformation.

[0099] Specifically, this step involves obtaining the Relay IR after operator fusion obtained in step (3-2), extracting all weight data from the Relay IR after operator fusion obtained in step (3-2), and performing format conversion on the weight data through bitwise operations and other methods according to the data layout format requirements of the NeuroRVcore operator library (4 bits per unit, aligned to 512 bits), rearranging it to conform to the data layout format supported by the NeuroRVcore operator library, and finally obtaining the Relay IR after data layout conversion.

[0100] (4) Using the LLVM backend code generator in the TVM infrastructure, the Relay IR after data layout transformation obtained in step (3) is lowered. Specifically, this step involves using the TVM infrastructure to compile each Relay operator node in the Relay IR after data layout transformation obtained in step (3) into the calling statement of the corresponding operator in the neuromorphic chip operator library, and combining all the compiled calling statements to finally obtain the LLVM intermediate code.

[0101] (5) Use the TVM infrastructure to compile the Relay IR after the data layout transformation obtained in step (3) to generate model files and weight files, and use the LLVM compiler that supports the NeuroRVcore backend to compile the LLVM intermediate code obtained in step (4) to generate operator files. Package the model files, weight files and operator files to generate the final deployment file package as the final SNN compilation result.

[0102] In this example, the model file typically describes the SNN model in JSON format; the weight file stores the static weight parameters in the SNN model; and the operator file contains the hardware-executable operators involved in the SNN model. These operators are compiled to adapt to the NeuroRVcore RISC-V instruction set and are usually organized as dynamic link libraries for the hardware to dynamically load and call during the inference phase. The above three together constitute a complete deployment package adapted to NeuroRVcore, which can run on NeuroRVcore to complete the inference execution of the SNN.

[0103] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for compiling a spiking neural network (SNN) for neuromorphic chips, characterized in that, Includes the following steps: (1) Obtain the neuromorphic chip operator library, register the corresponding Relay operator for each operator in the neuromorphic chip operator library based on the TVM infrastructure, and establish an association between the Relay operator and the corresponding operator in the neuromorphic chip operator library to obtain the Relay operator library adapted to the neuromorphic chip. (2) Obtain the SNN model described by the high-level language development framework, and perform model transformation processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain the Relay intermediate representation Relay IR which is equivalent to the SNN model; (3) The RelayIR obtained in step (2) is deformed to be oriented towards the SNN model and the characteristics of the neuromorphic chip, so as to obtain the deformed RelayIR; (4) Use the LLVM backend code generator in the TVM infrastructure to perform sinking processing on the Relay IR after data layout transformation obtained in step (3) to obtain LLVM intermediate code; (5) Use the TVM infrastructure to compile the Relay IR after the data layout transformation obtained in step (3) into model files and weight files, and use the LLVM compiler that supports the backend of neuromorphic chips to compile the LLVM intermediate code obtained in step (4) into operator files. Then, package the model files, weight files and operator files into the final deployment file package as the final SNN compilation result.

2. The method for compiling a spiking neural network (SNN) for neuromorphic chips according to claim 1, characterized in that, The neuromorphic chip operator library is a collection of operators optimized for neuromorphic chips.

3. The spiking neural network (SNN) compilation method for neuromorphic chips according to claim 1 or 2, characterized in that, Step (2) includes the following sub-steps: (2-1) Obtain the SNN model described by the high-level language development framework and convert it into a directed acyclic graph (DAG). The nodes in the DAG are the neuron clusters in the SNN model, and the edges in the DAG are the connections between the neuron clusters in the SNN model. (2-2) Use topological sorting to traverse all nodes in the DAG obtained in step (2-1) to obtain an unprocessed node N; (2-3) Obtain all input edges of node N in DAG obtained in step (2-2), and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set. (2-4) Determine whether the number of Relay operator nodes in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulating Relay operator node, use all Relay operator nodes in the Relay operator node set as input to the accumulating Relay operator node, and use the accumulating Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5). (2-5) Based on node N obtained in step (2-2), construct a leak-accumulate-excite LIF Relay operator node, and use the summary result obtained in step (2-4) as the input of the LIF Relay operator node. The LIF Relay operator node is the Relay operator node equivalent to node N. (2-6) For all remaining unprocessed nodes in the DAG, repeat steps (2-2) to (2-5) until all unprocessed nodes in the DAG have been processed, thus finally obtaining a Relay IR that is equivalent to the SNN model described by the high-level language development framework.

4. The spiking neural network (SNN) compilation method for neuromorphic chips according to claim 3, characterized in that, Step (2-3) specifically involves the following steps: First, for each input edge of node N obtained in step (2-2) in the DAG graph, obtain the synaptic type information of the input edge; then, based on the synaptic type information of the input edge, obtain the corresponding Relay operator in the Relay operator library, and construct the corresponding Relay operator node currNode based on the Relay operator; subsequently, obtain the starting point of the input edge in the DAG graph, i.e., the Relay operator node prevNode corresponding to the predecessor node, set the Relay operator node prevNode as the input of the Relay operator node currNode, and establish the dependency relationship between the Relay operator node prevNode and the Relay operator node currNode. Subsequently, for all remaining input edges corresponding to node N in the DAG graph, the above process is repeated, and all constructed Relay operator nodes are aggregated into a Relay operator node set.

5. The spiking neural network (SNN) compilation method for neuromorphic chips according to claim 4, characterized in that, Step (3) includes the following steps: (3-1) Obtain the neuromorphic chip operator library. Based on the fusion operator types in the neuromorphic chip operator library, summarize the SNN operator fusion modes supported by the neuromorphic chip. Based on the SNN operator fusion modes, register the SNN operator fusion modes and corresponding rewrite functions using the TVM infrastructure to obtain the SNN operator fusion mode library. (3-2) Use the SNN operator fusion pattern library obtained in step (3-1) to perform operator fusion processing on the Relay IR obtained in step (2) to obtain the Relay IR after operator fusion; (3-3) Perform data layout transformation on the RelayIR obtained by operator fusion in step (3-2) to obtain the Relay IR after data layout transformation.

6. The spiking neural network (SNN) compilation method for neuromorphic chips according to claim 5, characterized in that, Step (3-2) specifically involves: First, obtaining the SNN operator fusion pattern library defined in step (3-1), and using the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, using the MergeComposite optimization pass in the TVM infrastructure to perform subgraph matching processing on the RelayIR obtained in step (2) to obtain a Relay subgraph RSG that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; next, obtaining the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and using the rewrite function to rewrite the RSG into an operator-fused Relay subgraph; finally, adding a label to the operator-fused Relay subgraph to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate identification and processing in the subsequent code generation stage, thereby obtaining the operator-fused Relay IR.

7. The method for compiling a spiking neural network (SNN) for neuromorphic chips according to claim 6, characterized in that, Step (3-3) specifically involves first obtaining all weight data from the Relay IR obtained after operator fusion in step (3-2), and then converting all weight data into a format compatible with the neuromorphic chip operator library according to the data layout format requirements of the neuromorphic chip operator library, so as to obtain the Relay IR after data layout conversion.

8. The method for compiling a spiking neural network (SNN) for neuromorphic chips according to claim 7, characterized in that, Step (4) specifically involves using the LLVM backend code generator in the TVM infrastructure to compile each Relay operator node in the RelayIR after the data layout transformation obtained in step (3) into the corresponding operator call statement in the neuromorphic chip operator library, and then combining all the compiled call statements to obtain the LLVM intermediate code.

9. A spiking neural network (SNN) compilation system for neuromorphic chips, characterized in that, include: The first module is used to obtain the neuromorphic chip operator library, register the corresponding Relay operator for each operator in the neuromorphic chip operator library based on the TVM infrastructure, and establish an association between the Relay operator and the corresponding operator in the neuromorphic chip operator library to obtain a Relay operator library adapted to the neuromorphic chip. The second module is used to obtain the SNN model described by the high-level language development framework, and to perform model transformation processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in the first module, so as to obtain the Relay intermediate representation Relay IR which is equivalent to the SNN model. The third module is used to perform deformation processing on the RelayIR obtained from the second module in accordance with the SNN model and the characteristics of neuromorphic chips, so as to obtain the deformed RelayIR. The fourth module is used to use the LLVM backend code generator in the TVM infrastructure to perform downsinking processing on the Relay IR after the data layout transformation obtained in the third module, so as to obtain the LLVM intermediate code; The fifth module is used to compile the Relay IR obtained from the data layout transformation in the third module into model files and weight files using the TVM infrastructure, and to compile the LLVM intermediate code obtained from the fourth module into operator files using the LLVM compiler that supports the neuromorphic chip backend. Finally, the model files, weight files, and operator files are packaged into the final deployment file package as the final SNN compilation result.

Images