Brain-like chip-oriented spiking neural network compiling method and system

Abstract

The invention discloses a brain-like chip-oriented spiking neural network compiling method, which comprises the following steps of: expanding a spiking neural network Relay operator in a TVM, accessing and binding to a hardware operator library, automatically realizing intermediate representation conversion and optimization from a spiking neural network model to brain-like chip adaptation, and realizing compilation of a brain-like chip-oriented spiking neural network. In combination with optimization methods such as pulse neural network model-oriented operator fusion optimization and brain-like chip-oriented data layout conversion, the method is realized by fully utilizing high performance of a hardware operator library; and finally, generating a deployment file package based on a TVM infrastructure and an LLVM compiler, and efficiently completing compiling and deployment of the spiking neural network model on a brain-like chip. According to the invention, the technical problems of high learning cost and low development efficiency of the existing neural network compiling method based on DSL and the neural network compiling method based on hardware intra function or operator API can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and more specifically, relates to a spiking neural network (SNN) compilation method and system for brain-like chips. Background Art

[0002] Spiking Neural Network (SNN) has a natural advantage in processing time series data and optimizing energy efficiency by simulating the working mechanism of biological nerves. It can be used to implement complex artificial intelligence applications such as image classification and language recognition. The basic elements of SNN include spikes, neurons, and synapses, which together constitute the core mechanism for simulating biological nervous systems. SNN operators are usually used to describe the basic functional units of neuron and synapse computing behaviors, such as fully connected operators, convolutional connection operators, and leaky-integrate-fire (LIF) neuron operators.

[0003] Brain-like chips are based on SNN and simulate the information processing methods of the human brain through hardware circuits to realize perception, thinking, decision-making and other behaviors. They have the characteristics of low power consumption, integrated storage and computing, and high parallelism, showing broad application prospects in many fields.

[0004] Since brain-like chips are designed for SNN and usually have highly customized characteristics, there are many challenges in deploying SNN to brain-like chips. The existing methods for deploying neural network models to brain-like chips (i.e., neural network compilation methods) are mainly the following: the first is a neural network compilation method based on hardware intrinsic functions (compiler built-in functions) or operator application programming interfaces (Application Programming Interface, API for short), which manually implements model deployment by calling intrinsic functions or operator API interfaces; the second is a neural network compilation method based on domain-specific languages ​​(Domain-Specific Language, DSL for short), which is a programming language designed specifically for specific domains or hardware. DSL provides a high-level model abstract interface. By describing the high-level logic of the model, the DSL compiler automatically completes the underlying hardware optimization and deployment; the third method is a neural network compilation method based on intermediate representation (IR for short), which converts the neural network model into IR, and performs a series of hardware-independent and hardware-related optimizations based on IR, and finally generates hardware executable code.

[0005] However, the above neural network compilation methods all have some non-negligible defects:

[0006] First, the hardware design of brain-like chips is usually highly customized, with complicated details of neuron models, pulse and weight formats, and synaptic calculation rules. The existing neural network compilation methods based on hardware intrinsic functions or operator APIs require developers to have a deep understanding of the underlying hardware architecture of brain-like chips, which has a high learning cost and development threshold for developers, low development efficiency, prone to errors, and lack of scalability. The existing DSL-based neural network compilation methods also require learning new development languages ​​and programming paradigms. Although this lowers the development threshold, the learning cost is still high.

[0007] Second, the existing IR-based neural network compilation methods are mainly oriented towards artificial neural networks (ANN), lacking end-to-end compilation support for SNNs and brain-like chips. Mainstream neural network compilers such as the Tensor Virtual Machine (TVM) have not yet provided SNN compilation support, so there is a lack of optimization measures for the characteristics of SNNs and brain-like chips;

[0008] Third, designing and developing a DSL-based or IR-based neural network compiler from scratch requires comprehensive customization of the SNN and brain-like chip characteristics. The design is difficult, the R&D cycle is long, and the scalability and reusability are insufficient. Summary of the invention

[0009] In response to the above defects or improvement needs of the prior art, the present invention provides a pulse neural network compilation method and system for brain-like chips, which aims to solve the technical problems of high learning cost and low development efficiency in the existing DSL-based neural network compilation methods and hardware intrinsic function or operator API-based neural network compilation methods, as well as the technical problem that the existing IR-based neural network compilation methods lack optimization for SNN and brain-like chip characteristics, and the technical problem that the existing IR-based neural network compilation methods and DSL-based neural network compilation methods need to be customized from scratch for SNN and brain-like chips, and the development of dedicated neural network compilers is difficult, the research and development cycle is long, and the scalability and reusability are insufficient.

[0010] To achieve the above object, according to one aspect of the present invention, a spiking neural network (SNN) compilation method for a brain-like chip is provided, comprising the following steps:

[0011] (1) Obtain the brain-like chip operator library, register the corresponding Relay operator for each operator in the brain-like chip operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the brain-like chip operator library to obtain a Relay operator library adapted to the brain-like chip.

[0012] (2) Obtain the SNN model described by the high-level language development framework, and perform model conversion processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain a Relay intermediate representation (Relay IR) equivalent to the SNN model.

[0013] (3) The Relay IR obtained in step (2) is deformed according to the SNN model and the characteristics of the brain-like chip to obtain the deformed Relay IR.

[0014] (4) Use the LLVM backend code generator in the TVM infrastructure to sink the Relay IR after the data layout conversion obtained in step (3) to obtain the LLVM intermediate code.

[0015] (5) Use the TVM infrastructure to compile the Relay IR after the data layout conversion obtained in step (3) into a model file and a weight file, and use the LLVM compiler that supports the brain-like chip backend to compile the LLVM intermediate code obtained in step (4) into an operator file, and package the model file, weight file, and operator file into the final deployment file package as the final SNN compilation result.

[0016] Preferably, step (1) is to obtain a brain-like chip operator library from an existing brain-like chip development tool chain, wherein the brain-like chip operator library is a set of operators optimized for brain-like chips.

[0017] Preferably, step (2) comprises the following sub-steps:

[0018] (2-1) Obtaining the SNN model described by the high-level language development framework and converting it into a directed acyclic graph DAG, where the nodes in the DAG graph are the neuron clusters in the SNN model, and the edges in the DAG graph are the associations between the neuron clusters in the SNN model;

[0019] (2-2) Traverse all nodes in the DAG obtained in step (2-1) by topological sorting to obtain an unprocessed node N;

[0020] (2-3) Obtain all input edges of node N obtained in step (2-2) in the DAG, and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set.

[0021] (2-4) Determine whether the number of Relay operator nodes in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulated Relay operator node, use all Relay operator nodes in the Relay operator node set as inputs of the accumulated Relay operator node, and use the accumulated Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5);

[0022] (2-5) Based on the node N obtained in step (2-2), a leakage-accumulation-excitation LIF Relay operator node is constructed, and the summary result obtained in step (2-4) is used as the input of the LIF Relay operator node. The LIF Relay operator node is a Relay operator node equivalent to node N;

[0023] (2-6) For all remaining unprocessed nodes in the DAG, repeat the above steps (2-2) to (2-5) until all unprocessed nodes in the DAG are processed, thereby finally obtaining a Relay IR equivalent to the SNN model described by the high-level language development framework.

[0024] Preferably, step (2-3) is specifically as follows: first, for each input edge corresponding to the node N obtained in step (2-2) in the DAG graph, obtain the synaptic type information of the input edge; then, according to the synaptic type information of the input edge, obtain the corresponding Relay operator in the Relay operator library, and construct the corresponding Relay operator node currNode according to the Relay operator; then, obtain the Relay operator node prevNode corresponding to the starting point (i.e., the predecessor node) of the input edge in the DAG graph, use the Relay operator node prevNode as the input of the Relay operator node currNode, and establish a dependency relationship between the Relay operator node prevNode and the Relay operator node currNode; thereafter, for all remaining input edges corresponding to the node N in the DAG graph, repeat the above process, and aggregate all constructed Relay operator nodes into a Relay operator node set.

[0025] Preferably, step (3) comprises the following steps:

[0026] (3-1) Obtain the brain-like chip operator library, and summarize the SNN operator fusion mode supported by the brain-like chip according to the fusion operator type in the brain-like chip operator library; based on the SNN operator fusion mode, use the TVM infrastructure to register the SNN operator fusion mode and the corresponding rewrite function to obtain the SNN operator fusion mode library.

[0027] (3-2) Using the SNN operator fusion pattern library obtained in step (3-1), the Relay IR obtained in step (2) is subjected to operator fusion processing to obtain the Relay IR after operator fusion. The changes of the Relay IR before and after operator fusion are shown in the following example: Figure 4 shown.

[0028] (3-3) Perform data layout conversion on the Relay IR after operator fusion obtained in step (3-2) to obtain the Relay IR after data layout conversion.

[0029] Preferably, step (3-2) is specifically as follows: first, obtain the SNN operator fusion pattern library defined in step (3-1), and use the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, use the MergeComposite optimization in the TVM infrastructure to perform subgraph matching processing on the Relay IR obtained in step (2) to obtain a Relay subgraph RSG that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; then, obtain the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and use the rewrite function to rewrite RSG into a Relay subgraph after operator fusion; finally, add a label to the Relay subgraph after operator fusion to identify the SNN fusion mode to which the Relay subgraph belongs, so as to facilitate recognition and processing in the subsequent code generation stage, thereby obtaining the Relay IR after operator fusion.

[0030] Preferably, step (3-3) is specifically as follows: first, all weight data are obtained from the Relay IR after the operator fusion obtained in step (3-2); and according to the requirements of the brain-like chip operator library for the data layout format, all weight data are converted into a format compatible with the brain-like chip operator library to obtain the Relay IR after the data layout conversion.

[0031] Preferably, step (4) specifically comprises: using the LLVM backend code generator in the TVM infrastructure, compiling each Relay operator node in the Relay IR after the data layout conversion obtained in step (3) into a call statement of the corresponding operator in the brain-like chip operator library one by one, and combining all the compiled call statements to obtain the LLVM intermediate code.

[0032] According to another aspect of the present invention, a spiking neural network (SNN) compilation system for brain-like chips is provided, comprising:

[0033] The first module is used to obtain the brain-like chip operator library, register the corresponding Relay operator for each operator in the brain-like chip operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the brain-like chip operator library to obtain a Relay operator library adapted to the brain-like chip.

[0034] The second module is used to obtain the SNN model described by the high-level language development framework, and perform model conversion processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in the first module to obtain the Relay intermediate representation Relay IR equivalent to the SNN model.

[0035] The third module is used to deform the Relay IR obtained by the second module towards the SNN model and the characteristics of the brain-like chip to obtain the deformed Relay IR.

[0036] The fourth module is used to use the LLVM backend code generator in the TVM infrastructure to sink the Relay IR after the data layout conversion obtained in the third module to obtain the LLVM intermediate code.

[0037] The fifth module is used to use the TVM infrastructure to compile the Relay IR after the data layout conversion obtained in the third module into model files and weight files, and use the LLVM compiler that supports the brain-like chip backend to compile the LLVM intermediate code obtained in the fourth module into operator files, and package the model files, weight files and operator files into the final deployment file package as the final SNN compilation result.

[0038] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0039] (1) The present invention adopts steps (1) and (2), and by registering the SNN Relay operator in TVM and establishing an association with the operator in the brain-like chip operator library, the SNN model described by the high-level language development framework is connected to the TVM front end, thereby realizing TVM's compilation support for the SNN model. Therefore, the technical problems of high learning cost and low development efficiency existing in the existing neural network compilation methods based on DSL and the neural network compilation methods based on hardware intrinsic functions or operator APIs can be solved;

[0040] (2) Since the present invention adopts step (3), it adds IR transformation processing for SNN model and brain-like chip characteristics based on TVM infrastructure, thus solving the technical problem that the existing IR-based neural network compilation method lacks optimization for SNN and brain-like chip characteristics;

[0041] (3) The present invention adopts steps (1) to (5), which realizes an end-to-end compilation process from the SNN model described by the high-level language framework to the underlying machine code by reusing the high-level language framework, the brain-like chip operator library, the TVM infrastructure and the LLVM compiler, thereby solving the technical problems that the existing IR-based neural network compilation method and the DSL-based neural network compilation method need to be customized from scratch for SNN and brain-like chips, and the development of a dedicated neural network compiler is difficult, has a long research and development cycle, and lacks scalability and reusability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the pulse neural network compilation method for brain-like chips of the present invention;

[0043] Figure 2 is a flow chart of step (2) in the method of the present invention;

[0044] Figure 3 is a schematic diagram of step (2) in the method of the present invention;

[0045] Figure 4 It is a schematic diagram of Relay IR before and after operator fusion optimization of the present invention;

[0046] Figure 5 It is a schematic diagram of the weight data of the present invention before and after data layout conversion optimization;

[0047] Figure 6 It is a flow chart of code generation and basic compilation of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] Pulsing neural networks are believed to have higher biological interpretability and are better adapted to low-power brain-like chips. In theory, brain-like chips have the characteristics of low power consumption, high robustness, efficient parallelism, and adaptability. However, the current technical solutions have not yet been unified and are usually highly customized. The deployment of pulsing neural network models on brain-like chips has high learning costs and development difficulties, and there is a lack of end-to-end model compilation and deployment tools.

[0050] The basic idea of ​​the present invention is to provide a pulse neural network compilation method and system for brain-like chips based on the characteristics of the pulse neural network model. By extending the pulse neural network Relay operator in TVM, accessing and binding to the hardware operator library, the intermediate representation conversion and optimization from the pulse neural network model to the brain-like chip adaptation are automatically realized. Combined with the optimization methods such as operator fusion optimization for the pulse neural network model and data layout conversion for the brain-like chip, the high-performance implementation of the hardware operator library is fully utilized. Finally, the deployment file package is generated based on the TVM infrastructure and the LLVM compiler to efficiently complete the compilation and deployment of the pulse neural network model on the brain-like chip.

[0051] In order to make the purpose and features of the present invention more obvious and easy to understand, the present invention is further described in detail with reference to the accompanying drawings and specific implementation methods.

[0052] like Figure 1 As shown, the present invention provides a spiking neural network (SNN) compilation method for brain-like chips, comprising the following steps:

[0053] (1) Obtain the brain-like chip operator library, register the corresponding Relay operator for each operator in the brain-like chip operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the brain-like chip operator library to obtain a Relay operator library adapted to the brain-like chip.

[0054] Specifically, this step is to obtain a brain-like chip operator library from the existing brain-like chip development tool chain. The brain-like chip operator library is a set of operators optimized for brain-like chips, including multiple operator implementations customized for hardware architecture, covering commonly used operator implementations in pulse neural networks.

[0055] (2) Obtaining the SNN model described by the high-level language development framework, and performing model conversion processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain a Relay Intermediate Representation (Relay IR) equivalent to the SNN model;

[0056] like Figure 2 As shown, this step includes the following sub-steps:

[0057] (2-1) Obtaining the SNN model described by the high-level language development framework and converting it into a directed acyclic graph (DAG), wherein the nodes in the DAG are the neuron clusters in the SNN model, and the edges in the DAG are the associations between the neuron clusters in the SNN model;

[0058] (2-2) Traverse all nodes in the DAG obtained in step (2-1) by topological sorting to obtain an unprocessed node N;

[0059] (2-3) Obtain all input edges of node N obtained in step (2-2) in the DAG, and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set.

[0060] Specifically, this step is as follows: first, for each input edge corresponding to the node N obtained in step (2-2) in the DAG graph, obtain the synaptic type information of the input edge; then, according to the synaptic type information of the input edge, obtain the corresponding Relay operator in the Relay operator library, and construct the corresponding Relay operator node currNode according to the Relay operator; then, obtain the Relay operator node prevNode corresponding to the starting point (i.e., the predecessor node) of the input edge in the DAG graph, use the Relay operator node prevNode as the input of the Relay operator node currNode, and establish a dependency relationship between the Relay operator node prevNode and the Relay operator node currNode; then, for all remaining input edges corresponding to the node N in the DAG graph, repeat the above process, and aggregate all constructed Relay operator nodes into a Relay operator node set for subsequent node conversion;

[0061] (2-4) Determine whether the number of Relay operator nodes in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulation Relay operator node, use all Relay operator nodes in the Relay operator node set as inputs of the accumulation Relay operator node, and use the accumulation Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5);

[0062] (2-5) Based on the node N obtained in step (2-2), a Leaky-Integrate-Fire (LIF) Relay operator node is constructed, and the summary result obtained in step (2-4) is used as the input of the LIF Relay operator node. The LIF Relay operator node is a Relay operator node equivalent to the node N;

[0063] (2-6) For all remaining unprocessed nodes in the DAG, repeat the above steps (2-2) to (2-5) until all unprocessed nodes in the DAG are processed, thereby finally obtaining a Relay IR equivalent to the SNN model described by the high-level language development framework. An example of converting the SNN model to Relay IR is shown in the following figure. Figure 3 shown.

[0064] The advantages of the above steps (1) to (2) are that the dedicated Relay operator in the SNN field is registered based on the TVM infrastructure and associated with the brain-like chip operator library, and the SNN described by the high-level language development framework is converted into Relay IR, so that developers do not need to deeply understand the underlying hardware architecture and can directly use the high-level language development framework to develop SNN models, which greatly reduces the learning cost and development threshold.

[0065] (3) The Relay IR obtained in step (2) is transformed according to the SNN model and the characteristics of the brain-like chip to obtain the transformed Relay IR.

[0066] This step (3) includes the following steps:

[0067] (3-1) Obtain the brain-like chip operator library, and summarize the SNN operator fusion mode supported by the brain-like chip according to the fusion operator type in the brain-like chip operator library; based on the SNN operator fusion mode, use the TVM infrastructure to register the SNN operator fusion mode and the corresponding rewrite function to obtain the SNN operator fusion mode library.

[0068] Specifically, the brain-like chip operator library described in this step contains multiple efficient fusion operators customized for brain-like chips. By analyzing the fusion operators in the brain-like chip operator library, the SNN operator fusion mode supported by the brain-like chip is extracted. In this example, the SNN operator fusion modes include: fully connected-LIF mode, convolutional connection-LIF mode, maximum pooling connection-LIF mode, etc.

[0069] (3-2) Using the SNN operator fusion pattern library obtained in step (3-1), the Relay IR obtained in step (2) is subjected to operator fusion processing to obtain the Relay IR after operator fusion. The changes of the Relay IR before and after operator fusion are shown in the following example: Figure 4 shown.

[0070] Specifically, this step is as follows: first, obtain the SNN operator fusion pattern library defined in step (3-1), and use the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, use the MergeComposite optimization pass in the TVM infrastructure to perform subgraph matching on the Relay IR obtained in step (2) to obtain a Relay subgraph (RSG) that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; then, obtain the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and use the rewrite function to rewrite the RSG into the Relay subgraph after operator fusion; finally, add a tag to the Relay subgraph after operator fusion to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate recognition and processing in the subsequent code generation stage, thereby obtaining the Relay IR after operator fusion;

[0071] (3-3) Perform data layout conversion on the Relay IR after operator fusion obtained in step (3-2) to obtain the Relay IR after data layout conversion.

[0072] Specifically, this step is as follows: first, all weight data is obtained from the Relay IR after the operator fusion obtained in step (3-2). According to the requirements of the brain-like chip operator library for the data layout format, all weight data is converted into a format compatible with the brain-like chip operator library to obtain the Relay IR after the data layout conversion. The changes before and after the weight data layout conversion are shown in the following example. Figure 5 shown.

[0073] The advantage of this step (3) is that by adding the SNN operator fusion mode supported by the brain-like chip in TVM, the subgraph matching mechanism of TVM is used to automatically realize the matching and rewriting of the Relay subgraph, so that multiple operators are merged into an efficient fusion operator, and the efficient operator implementation of the brain-like chip hardware operator library is fully utilized; by adding the data layout conversion optimization process, the weight data format is adjusted to the layout format supported by the brain-like chip operator library, avoiding the additional overhead caused by data format mismatch, and solving the problem that the existing neural network compiler lacks optimization for SNN and brain-like chip characteristics;

[0074] (4) Use the LLVM backend code generator in the TVM infrastructure to lower the Relay IR after the data layout conversion obtained in step (3) to obtain the LLVM intermediate code.

[0075] Specifically, this step uses the LLVM backend code generator in the TVM infrastructure to compile each Relay operator node in the Relay IR after the data layout conversion obtained in step (3) into a call statement of the corresponding operator in the brain-like chip operator library, and combines all the compiled call statements to obtain the LLVM intermediate code.

[0076] (5) Use the TVM infrastructure to compile the Relay IR after the data layout conversion obtained in step (3) into a model file and a weight file, and use the LLVM compiler that supports the brain-like chip backend to compile the LLVM intermediate code obtained in step (4) into an operator file, and package the model file, weight file, and operator file into the final deployment file package as the final SNN compilation result. The process of step (4) and step (5) is as follows: Figure 6 shown.

[0077] The advantage of the above steps (1) to (5) is that by reusing the high-level language framework and brain-like chip operator library and combining the existing compilation infrastructure of TVM and LLVM, the high cost and complexity of building a compiler from scratch are effectively avoided; by establishing a fully automated end-to-end compilation process from high-level language framework to underlying hardware code generation, the workload of developers in model conversion, optimization and deployment is reduced, and the development cycle is greatly shortened.

[0078] Example

[0079] The brain-like chip platform involved in this embodiment is a brain-like chip NeuroRVcore based on the RISC-V instruction set. Specifically, NeuroRVcore is based on the RISC-V microprocessor to expand the brain-like computing microarchitecture and brain-like computing instruction set, supports parallel computing of up to 1024 configurable LIF neurons, and supports synaptic computing of various granularities, with low power consumption and strong scalability. The brain-like chip platform supports model development based on the brain-like chip operator library (hereinafter referred to as "NeuroRVcore operator library") and supports C / C++ and LLVM code compilation.

[0080] This embodiment is used to further illustrate the present invention. The following are the specific implementation steps of Embodiment 2:

[0081] (1) Obtain the NeuroRVcore operator library from the development tool chain supporting NeuroRVcore, register the Relay operator for each operator in the NeuroRVcore operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the brain-like chip operator library to obtain the Relay operator library adapted to NeuroRVcore. Specifically, the NeuroRVcore operator library contains multiple high-performance operator implementations for the NeuroRVcore brain-like chip. The NeuroRVcore operator library includes but is not limited to the following operators: fully connected operator, accumulation operator, LIF operator, fully connected-LIF fusion operator;

[0082] (2) Obtaining the SNN model described by the high-level language development framework BindsNet, and performing model conversion processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain a RelayIR equivalent to the SNN model;

[0083] This step specifically includes the following steps:

[0084] (2-1) Obtain the SNN model described by the high-level language development framework BindsNet and convert it into DAG.

[0085] Specifically, based on the source code of the SNN model described by BindsNet, the Python interpreter is used to obtain the Network class object in the BindsNet high-level language development framework. The Network class object stores the information of the user-defined SNN model. By obtaining the Network class object, the DAG composed of the SNN model can be obtained; specifically, the layers data structure in the Network stores the information of the neuron clusters, and the connections data structure in the Network stores the information of the connections between the neuron clusters;

[0086] (2-2) traverse all nodes in the DAG obtained in step (2-1) in topological order to obtain an unprocessed node N;

[0087] (2-3) Obtain all input edges of node N obtained in step (2-2) in the DAG, and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set.

[0088] Specifically, if the synaptic type information of a certain input edge of node N is fully connected, then obtain the fully connected Relay operator from the Relay operator library and construct a fully connected Relay operator node FCNode (for other synaptic types, map to the corresponding Relay operator in the Relay operator library, which will not be repeated here); then obtain the Relay operator node LIFNode corresponding to the starting point (i.e., the predecessor node) of the input edge in the DAG graph, and use LIFNode as the input of FCNode to establish a dependency relationship between LIFNode and FCNode. Finally, for all remaining input edges corresponding to node N in the DAG graph, repeat the above process, and summarize all constructed Relay operator nodes into a Relay operator node set for subsequent node conversion;

[0089] (2-4) Determine whether the number of elements in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulation Relay operator node, use all Relay operator nodes in the Relay operator node set as inputs of the accumulation Relay operator node, and use the accumulation Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5);

[0090] (2-5) Obtain the node N obtained in step (2-2), construct a LIF Relay operator node, use the summary result obtained in step (2-4) as the input of the LIF Relay operator node, and obtain a Relay operator node equivalent to node N;

[0091] (2-6) For all remaining unprocessed nodes in the DAG, repeat the above steps (2-2) to (2-5) until all unprocessed nodes in the DAG are processed, thereby finally obtaining a Relay IR equivalent to the SNN model described by the high-level language development framework.

[0092] (3) The Relay IR obtained in step (2) is transformed according to the SNN model and NeuroRVcore characteristics to obtain the transformed Relay IR.

[0093] Specifically, this step includes the following sub-steps:

[0094] (3-1) Obtain the NeuroRVcore operator library, obtain the SNN operator fusion mode supported by the brain-like chip according to the fusion operator type in the NeuroRVcore operator library, and use the TVM infrastructure to register the rewrite function (Rewrite Function) corresponding to the SNN operator fusion mode to obtain the SNN operator fusion mode library.

[0095] Specifically, the NeuroRVcore operator library described in this step contains multiple efficient fusion operators customized for NeuroRVcore. By analyzing the fusion operators in the brain-like chip operator library, the SNN operator fusion mode supported by the brain-like chip is extracted. In this example, the SNN operator fusion mode includes but is not limited to the fully connected-LIF mode.

[0096] (3-2) Use the SNN operator fusion pattern library obtained in step (3-1) to perform SNN operator fusion processing on the Relay IR obtained in step (2) to obtain the Relay IR after operator fusion.

[0097] Specifically, this step is as follows: first, obtain the SNN operator fusion pattern library defined in step (3-1), and use the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, use the MergeComposite optimization pass in the TVM infrastructure to perform subgraph matching on the Relay IR obtained in step (2) to obtain a Relay subgraph (RSG) that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; then, obtain the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and use the rewrite function to rewrite the RSG into the Relay subgraph after operator fusion; finally, add a tag to the Relay subgraph after operator fusion to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate recognition and processing in the subsequent code generation stage, thereby obtaining the Relay IR after operator fusion;

[0098] (3-3) Perform data layout transformation and deformation processing on the Relay IR after operator fusion obtained in step (3-2) to obtain the Relay IR after data layout transformation.

[0099] Specifically, this step is to obtain the Relay IR after the operator fusion obtained in step (3-2), extract all weight data in the Relay IR after the operator fusion obtained in step (3-2), and convert the weight data format through bit operations and other operations according to the requirements of the NeuroRVcore operator library on the data layout format (4 bits as a unit and aligned to 512 bits), and rearrange it into a data layout format that conforms to the NeuroRVcore operator library supported, and finally obtain the Relay IR after the data layout conversion.

[0100] (4) Use the LLVM backend code generator in the TVM infrastructure to lower the Relay IR after the data layout conversion obtained in step (3). Specifically, this step is to use the TVM infrastructure to compile each Relay operator node in the Relay IR after the data layout conversion obtained in step (3) into the call statement of the corresponding operator in the brain-like chip operator library one by one, and combine all the compiled call statements to finally obtain the LLVM intermediate code.

[0101] (5) Use the TVM infrastructure to compile the Relay IR after the data layout conversion obtained in step (3) to generate model files and weight files, and use the LLVM compiler that supports the NeuroRVcore backend to compile the LLVM intermediate code obtained in step (4) to generate operator files. Package the model file, weight file, and operator file to generate the final deployment file package as the final SNN compilation result.

[0102] In this example, the model file usually describes the SNN model in JSON format; the weight file stores the static weight parameters in the SNN model; the operator file contains the hardware executable operators involved in the SNN model. These operators are compiled to adapt to the RISC-V instruction set of NeuroRVcore and are usually organized in the form of dynamic link libraries for dynamic loading and calling by the hardware during the inference phase. The above three together constitute a complete deployment file package adapted to NeuroRVcore, which can run on NeuroRVcore to complete the inference execution of SNN.

[0103] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A spiking neural network (SNN) compilation method for brain-like chips, characterized in that: The following steps are involved: (1) Obtain the brain-like chip operator library, register the corresponding Relay operator for each operator in the brain-like chip operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the brain-like chip operator library to obtain a Relay operator library adapted to the brain-like chip. (2) Obtain the SNN model described by the high-level language development framework, and perform model conversion processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in step (1) to obtain a Relay intermediate representation (Relay IR) equivalent to the SNN model. (3) The Relay IR obtained in step (2) is deformed according to the SNN model and the characteristics of the brain-like chip to obtain the deformed Relay IR. (4) Use the LLVM backend code generator in the TVM infrastructure to sink the Relay IR after the data layout conversion obtained in step (3) to obtain the LLVM intermediate code. (5) Use the TVM infrastructure to compile the Relay IR after the data layout conversion obtained in step (3) into a model file and a weight file, and use the LLVM compiler that supports the brain-like chip backend to compile the LLVM intermediate code obtained in step (4) into an operator file, and package the model file, weight file, and operator file into the final deployment file package as the final SNN compilation result.

2. The spiking neural network (SNN) compilation method for brain-like chips according to claim 1 is characterized in that: Step (1) is to obtain a brain-like chip operator library from an existing brain-like chip development tool chain, wherein the brain-like chip operator library is a set of operators optimized for brain-like chips.

3. The spiking neural network (SNN) compilation method for brain-like chips according to claim 1 or 2, characterized in that: Step (2) includes the following sub-steps: (2-1) Obtaining the SNN model described by the high-level language development framework and converting it into a directed acyclic graph DAG, where the nodes in the DAG graph are the neuron clusters in the SNN model, and the edges in the DAG graph are the associations between the neuron clusters in the SNN model; (2-2) Traverse all nodes in the DAG obtained in step (2-1) by topological sorting to obtain an unprocessed node N; (2-3) Obtain all input edges of node N obtained in step (2-2) in the DAG, and convert each input edge into a corresponding Relay operator node according to the Relay operator library obtained in step (1). The Relay operator nodes corresponding to all input edges constitute a Relay operator node set. (2-4) Determine whether the number of Relay operator nodes in the Relay operator node set obtained in step (2-3) is greater than 1 or equal to 1. If it is greater than 1, generate an accumulated Relay operator node, use all Relay operator nodes in the Relay operator node set as inputs of the accumulated Relay operator node, and use the accumulated Relay operator node as the summary result, and then proceed to step (2-5); if it is equal to 1, use the Relay operator node as the summary result, and then proceed to step (2-5); (2-5) Based on the node N obtained in step (2-2), a leakage-accumulation-excitation LIF Relay operator node is constructed, and the summary result obtained in step (2-4) is used as the input of the LIF Relay operator node. The LIF Relay operator node is a Relay operator node equivalent to node N; (2-6) For all remaining unprocessed nodes in the DAG, repeat the above steps (2-2) to (2-5) until all unprocessed nodes in the DAG are processed, thereby finally obtaining a Relay IR equivalent to the SNN model described by the high-level language development framework.

4. The spiking neural network (SNN) compilation method for brain-like chips according to any one of claims 1 to 3, characterized in that: Step (2-3) is specifically as follows: first, for each input edge corresponding to the node N obtained in step (2-2) in the DAG graph, obtain the synaptic type information of the input edge; then, according to the synaptic type information of the input edge, obtain the corresponding Relay operator in the Relay operator library, and construct the corresponding Relay operator node currNode according to the Relay operator; then, obtain the Relay operator node prevNode corresponding to the starting point (i.e., the predecessor node) of the input edge in the DAG graph, use the Relay operator node prevNode as the input of the Relay operator node currNode, and establish a dependency relationship between the Relay operator node prevNode and the Relay operator node currNode; Thereafter, for all remaining input edges corresponding to node N in the DAG graph, the above process is repeated, and all constructed Relay operator nodes are aggregated into a Relay operator node set.

5. The spiking neural network (SNN) compilation method for brain-like chips according to claim 4 is characterized in that: Step (3) comprises the following steps: (3-1) Obtain the brain-like chip operator library, and summarize the SNN operator fusion mode supported by the brain-like chip according to the fusion operator type in the brain-like chip operator library; based on the SNN operator fusion mode, use the TVM infrastructure to register the SNN operator fusion mode and the corresponding rewrite function to obtain the SNN operator fusion mode library. (3-2) Using the SNN operator fusion pattern library obtained in step (3-1), perform operator fusion processing on the Relay IR obtained in step (2) to obtain the Relay IR after operator fusion. An example of the change of Relay IR before and after operator fusion is shown in FIG4 . (3-3) Perform data layout conversion on the Relay IR after operator fusion obtained in step (3-2) to obtain the Relay IR after data layout conversion.

6. The spiking neural network (SNN) compilation method for brain-like chips according to claim 5 is characterized in that: Step (3-2) is specifically as follows: first, obtain the SNN operator fusion pattern library defined in step (3-1), and use the TVM infrastructure to parse and load the SNN operator fusion pattern library; then, use the MergeComposite optimization in the TVM infrastructure to perform subgraph matching processing on the Relay IR obtained in step (2) to obtain a Relay subgraph RSG that conforms to any SNN operator fusion pattern in the SNN operator fusion pattern library; then, obtain the corresponding rewrite function of the RSG from the SNN operator fusion pattern library, and use the rewrite function to rewrite RSG into a Relay subgraph after operator fusion; finally, add a label to the Relay subgraph after operator fusion to identify the SNN fusion pattern to which the Relay subgraph belongs, so as to facilitate recognition and processing in the subsequent code generation stage, thereby obtaining the Relay IR after operator fusion.

7. The spiking neural network (SNN) compilation method for brain-like chips according to claim 6 is characterized in that: Specifically, step (3-3) is as follows: first, all weight data are obtained from the Relay IR after the operator fusion obtained in step (3-2); and according to the requirements of the brain-like chip operator library for the data layout format, all weight data are converted into a format compatible with the brain-like chip operator library to obtain the Relay IR after the data layout conversion.

8. The method for compiling a spiking neural network (SNN) for a brain-like chip according to claim 7, characterized in that: Step (4) specifically includes using the LLVM backend code generator in the TVM infrastructure to compile each Relay operator node in the Relay IR after the data layout conversion obtained in step (3) into a call statement of the corresponding operator in the brain-like chip operator library, and combine all the compiled call statements to obtain the LLVM intermediate code.

9. A spiking neural network (SNN) compilation system for brain-like chips, characterized in that: include: The first module is used to obtain the brain-like chip operator library, register the corresponding Relay operator for each operator in the brain-like chip operator library based on the TVM infrastructure, and associate the Relay operator with the corresponding operator in the brain-like chip operator library to obtain a Relay operator library adapted to the brain-like chip. The second module is used to obtain the SNN model described by the high-level language development framework, and perform model conversion processing on the SNN model described by the high-level language development framework based on the Relay operator library obtained in the first module to obtain the Relay intermediate representation Relay IR equivalent to the SNN model. The third module is used to deform the Relay IR obtained by the second module towards the SNN model and the characteristics of the brain-like chip to obtain the deformed Relay IR. The fourth module is used to use the LLVM backend code generator in the TVM infrastructure to sink the Relay IR after the data layout conversion obtained in the third module to obtain the LLVM intermediate code. The fifth module is used to use the TVM infrastructure to compile the Relay IR after the data layout conversion obtained in the third module into model files and weight files, and use the LLVM compiler that supports the brain-like chip backend to compile the LLVM intermediate code obtained in the fourth module into operator files, and package the model files, weight files and operator files into the final deployment file package as the final SNN compilation result.

Images