Neural network hardware acceleration scheduling algorithm based on directed graph representation
By using a hardware acceleration scheduling algorithm for neural networks based on directed graph representation, the deployment challenge of complex network architectures in neural network accelerators is solved, achieving unified expression of temporal dependencies and efficient resource utilization, thereby improving computational and energy efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2021-07-27
- Publication Date
- 2026-04-28
AI Technical Summary
Existing neural network accelerators face unpredictable and irregular connectivity issues when deploying complex network architectures, especially lacking a unified expression in time-dependent modules, and having low on-chip cache resource utilization efficiency, resulting in low energy efficiency.
A hardware-accelerated scheduling algorithm based on directed graph representation is adopted. By using graph theory modeling, operators are treated as nodes and dependencies as edges to construct a directed graph. The algorithm searches the graph computation path to reduce data transmission and dynamically allocates on-chip cache resources to optimize the computation order and cache utilization.
It achieves a unified representation of various neural networks and efficient resource utilization, improving the accelerator's computational and energy efficiency while reducing latency and power consumption.
Smart Images
Figure CN115687236B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of neural network technology, specifically relating to a hardware-accelerated scheduling algorithm for neural networks based on directed graph representation. Background Technology
[0002] The evolution of network topology has been a driving force behind the development of neural networks. Currently, novel connection architectures for neural networks have become a research focus, evolving from chain-like connections to residual blocks, initial structures, and dense connections. These innovative topological results have yielded excellent performance on many tasks. Recently, Neural Architecture Search (NAS) frameworks have been proposed to find network structures with better expressive power or fewer network parameters. However, the automatic search process of NAS leads to unpredictable network topology and more irregular connections between layers. This unpredictability and irregularity pose significant challenges to the deployment of networks on accelerators. There is an urgent need for a scheduling strategy that can handle various network architectures, enabling more efficient utilization of accelerator's dedicated computing arrays and on-chip memory resources.
[0003] First, the connectivity complexity of emerging networks is not only reflected in the long distances and multiple dependencies in connections, but also in the temporal dependencies between computational layers. Some works have implemented graph representations of connections between convolutional neural network (CNN) layers, but recurrent neural network (LSTM), recurrent gate units (GRU), attention modules, and other neural network modules all have many temporal dependencies, which have not yet been unified and expressed in a unified manner.
[0004] In addition, due to the limited on-chip cache resources on dedicated accelerators, the feature maps of the model often need to be transferred between the accelerator's on-chip cache and external memory, which introduces additional latency and reduces energy efficiency. Summary of the Invention
[0005] This invention is made to solve the above-mentioned problems, and its purpose is to provide a hardware-accelerated scheduling algorithm for neural networks based on directed graph representation. The technical solution adopted by this invention is as follows:
[0006] This invention provides a hardware-accelerated scheduling algorithm for neural networks based on directed graph representation, characterized by the following steps: Step S1, modeling the target neural network based on graph theory, where operators are represented using nodes, operator dependencies are represented using edges, and the temporal nature of dependencies is represented by edge weights, thus obtaining a directed graph of the target neural network; Step S2, with the aim of reducing on-chip data transmission, searching for graph computation paths in the directed graph based on the dependencies of nodes in the directed graph, and obtaining search results; Step S3, with the aim of improving the efficiency of on-chip caching, dynamically allocating on-chip cache resources based on graph computation paths and the dependencies of the directed graph.
[0007] The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation provided by this invention may also have the following technical features: the operator has operator input and operator output, the operator dependency is used to describe the direction of the operator input and operator output, the direction is represented by directed edges, the direction of the operator input is the in edge, and the direction of the operator output is the out edge.
[0008] The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation provided by this invention may also have the following technical features: the search results are placed into the processing queue of the operator in order; for each node, it is determined whether the node's predecessor node has been added to the processing queue; if the determination is yes, the node is added to the processing queue; if the determination is no, the predecessor nodes are traversed and the addition of predecessor nodes is performed iteratively.
[0009] The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation provided by this invention may also have the following technical features: the search results are placed into the processing queue of the operator in order; for each node, it is determined whether the condition of the node is met and whether the node has been added to the processing queue; when the condition is met, it is checked and determined whether there are subsequent nodes of the node that have met the conditions; when the condition is met, the node and its subsequent nodes are fused and calculated.
[0010] The hardware acceleration scheduling algorithm for neural networks based on directed graph representation provided by this invention may also have the following technical features: the operator input includes the size of the cache, the directed graph of the target neural network, and the dimension of a single data input to the target neural network; the operator output includes the size of the input data that can be batch-processed and the address allocation of each operator output stored in the cache during the calculation process.
[0011] The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation provided by this invention may also have the following technical features: for each node, it is determined whether the node is merged with other nodes. When the determination is yes, the last operator after merging outputs the calculation result of the node to the on-chip cache for temporary storage; when the determination is no, the calculation result of the node is output to the on-chip cache for temporary storage.
[0012] The neural network hardware acceleration scheduling algorithm based on directed graph representation provided by this invention can also have the following technical feature: when a node completes its computation, a search is performed in the directed graph, and the on-chip cache occupied by nodes that are no longer dependent on by subsequent nodes to be computed is released.
[0013] The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation provided by this invention may also have the following technical features, wherein the operators include convolution, fully connected, batch normalization, bias processing, vector multiplication, vector addition, and activation functions.
[0014] Invention Function and Effect
[0015] The hardware acceleration scheduling algorithm for neural networks based on directed graph representation according to the present invention models the target neural network based on graph theory to obtain the directed graph representation of the neural network. This novel neural network representation method realizes a unified representation of neural networks with time-dependent characteristics, can well represent various existing neural networks, and has good scalability, and can be compatible with various new neural network structures that may emerge in the future.
[0016] Furthermore, based on this directed graph, a graph computation path search algorithm can minimize data movement between the computation module and the on-chip storage module. Moreover, based on the graph computation paths and the dependencies within the directed graph, non-dependent results from the on-chip cache can be dynamically released, meaning on-chip cache resources can be dynamically allocated, significantly improving on-chip cache utilization efficiency and enabling batch processing of more data. The combination of these two approaches enhances the versatility of neural network accelerators, improves their computational efficiency, and reduces power consumption and latency in the accelerator hardware. Attached Figure Description
[0017] Figure 1 This is a flowchart of a hardware-accelerated scheduling algorithm for neural networks based on directed graph representation, according to an embodiment of the present invention.
[0018] Figure 2 This is a schematic diagram of a graph representation of a neural network according to an embodiment of the present invention;
[0019] Figure 3 This is an example diagram of graph computation path search according to an embodiment of the present invention. Detailed Implementation
[0020] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes in detail the hardware acceleration scheduling algorithm for neural networks based on directed graph representation.
[0021] <Example>
[0022] Figure 1 This is a flowchart of a hardware-accelerated scheduling algorithm for neural networks based on directed graph representation, according to an embodiment of the present invention.
[0023] like Figure 1 As shown, the hardware-accelerated scheduling algorithm for neural networks based on directed graph representation includes the following steps:
[0024] Step S1: Model the target neural network based on graph theory. Operators are represented by nodes, operator dependencies are represented by edges, and the temporal nature of dependencies is represented by edge weights, thus obtaining a directed graph of the target neural network.
[0025] Figure 2 This is a schematic diagram of a graph representation of a neural network according to an embodiment of the present invention.
[0026] like Figure 2 As shown, preferably, the operators of the neural network are represented using graph nodes, and the operators include input, fully connected, convolution, normalization layer, bias layer, activation function layer, vector multiplication, vector addition, and other operators.
[0027] Preferably, the input-output relationship of a neural network is represented using directed edges: the incoming edges of an operator represent its input source, and the outgoing edges represent its output direction. An operator can have multiple incoming edges and multiple outgoing edges.
[0028] Step S2, with the aim of reducing on-chip data transmission, searches for graph computation paths in the directed graph based on the dependencies of nodes in the directed graph, and obtains the search results.
[0029] On-chip transfer refers to data transfer within the neural network accelerator chip. In popular neural network accelerators, on-chip caches are commonly designed to reduce on-chip and off-chip data interaction. However, data transfer between the on-chip computing module and the cache module is also a significant source of power consumption. To reduce this data interaction, during computation, we avoid transferring data out of the computing module as much as possible for continuously computed units, performing continuous computation before transferring the final result from the computing module to the cache module. This approach minimizes the number of times data is moved between computation and storage.
[0030] Figure 3 This is an example diagram of graph computation path search according to an embodiment of the present invention.
[0031] like Figure 3 As shown, based on the directed graph constructed in step S1, the operators are computed and sorted according to the dependencies of the nodes. The operators are computed and sorted according to the following sorting principles:
[0032] If the preceding nodes of the current node have not been fully added to the computation queue, then add the preceding nodes to the computation queue first.
[0033] If there is a post-processing operator in the subsequent nodes of the current node, add the current node to the computation queue and set the node's result not to be output to the computation module, and then add the post-processing operator immediately after the current node to the computation queue.
[0034] If there are no post-processing operators in the subsequent nodes of the current node, add the current node to the computation queue and set the node's result as the output to the cache module;
[0035] The computation is complete if all nodes have been added to the computation queue.
[0036] Step S3, with the aim of improving the efficiency of on-chip cache, dynamically allocates on-chip cache resources based on graph computation paths and the dependencies of directed graphs, combined with some hardware parameters.
[0037] In this embodiment, the efficiency of the on-chip cache is defined as the maximum amount of data that the on-chip cache can process without interacting with off-chip storage.
[0038] In neural network accelerators, on-chip caches are typically divided into two parts: a weight cache module responsible for storing neural network weights, and a feature map cache module responsible for storing input data, intermediate network computation results, and final network computation results.
[0039] For the weight caching module, since the amount of weight data is fixed, only a static on-chip cache address allocation is needed.
[0040] For the feature map caching module, since the intermediate results that need to be cached are constantly changing, and in a neural network computation graph composed of multiple operators, the computation results of some of the earlier operators are no longer needed after a certain point in time, the space they occupy can be released to store more computational data. This process is called dynamic allocation of cache space.
[0041] Therefore, improving the efficiency of on-chip cache mainly refers to improving the efficiency of the feature map cache module.
[0042] The dynamic allocation algorithm for on-chip cache resources in step S3 consists of three steps:
[0043] Step S3-1: Dynamically allocate output addresses based on the sorting of computing nodes in step S2;
[0044] Step S3-2: Determine whether the result of the current computing node needs to be output to the cache module. If the determination is yes, find the first address that matches the output size in the feature map cache module and allocate it. If the determination is no, no address allocation is required.
[0045] Step S3-3: After the current computing node finishes outputting its results, a search is performed throughout the entire computing graph. If a result stored in the cache is no longer needed by subsequent uncomputed nodes, its space in the cache can be released, thereby achieving dynamic allocation of on-chip cache resources.
[0046] As described above, the target neural network is represented by a directed graph, and the operator computation order and intermediate results during the computation process are obtained. Combined with some hardware parameters, the on-chip data transfer is reduced and the on-chip cache resources are dynamically allocated, thereby improving computational efficiency and reducing hardware power consumption and latency.
[0047] Functions and effects of the embodiments
[0048] According to the directed graph representation-based neural network hardware acceleration scheduling algorithm provided in this embodiment, the target neural network is modeled based on graph theory to obtain the directed graph representation of the neural network. This novel neural network representation method realizes a unified representation of neural networks with time-dependent characteristics, can well represent various existing neural networks, and has good scalability, and can be compatible with various new neural network structures that may emerge in the future.
[0049] Furthermore, based on this directed graph, a graph computation path search algorithm can minimize data movement between the computation module and the on-chip storage module. Moreover, based on the graph computation paths and the dependencies within the directed graph, non-dependent results from the on-chip cache can be dynamically released, meaning on-chip cache resources can be dynamically allocated, significantly improving on-chip cache utilization efficiency and enabling batch processing of more data. The combination of these two approaches enhances the versatility of neural network accelerators, improves their computational efficiency, and reduces power consumption and latency in the accelerator hardware.
[0050] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.
Claims
1. A hardware-accelerated scheduling algorithm for neural networks based on directed graph representation, characterized in that, include: Step S1: Model the target neural network based on graph theory. Operators are represented by nodes, operator dependencies are represented by edges, and the temporal nature of the operator dependencies is represented by the weights of the edges, thus obtaining a directed graph of the target neural network. Step S2: To reduce on-chip data transmission, a graph computation path is searched for the directed graph based on the dependencies of the nodes in the directed graph to obtain the search results. Step S3, with the aim of improving the efficiency of the on-chip cache, dynamically allocates the resources of the on-chip cache based on the graph computation path and the dependencies of the directed graph. The search results are sequentially placed into the processing queue of the operator. For each node, it is determined whether its preceding node has been added to the processing queue. If yes, the node is added to the processing queue; otherwise, the preceding nodes are traversed and the process of adding preceding nodes is repeated iteratively. The search results are placed sequentially into the processing queue of the operator. For each node, it is determined whether the conditions of the node are met and whether the node has been added to the processing queue. If the conditions are met, it is checked whether there are any subsequent nodes of the node that have met the conditions. If the conditions are met, the node and the subsequent nodes are merged and calculated. For each node, it is determined whether the node is merged with other nodes. If the determination is yes, the last operator after merging outputs the calculation result of the node to the on-chip cache for temporary storage. If the determination is negative, the calculation result of the node is output to the on-chip cache for temporary storage. When the calculation of a node is completed, a search is performed in the directed graph to release the on-chip cache occupied by the node that is no longer depended on by the subsequent nodes to be calculated.
2. The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation according to claim 1, characterized in that: in, The operator has operator input and operator output. The operator dependency describes the direction of the operator input and the operator output, and the direction is represented by directed edges. The direction of the operator input is the incoming edge, and the direction of the operator output is the outgoing edge.
3. The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation according to claim 1, characterized in that: in, The operator input includes the size of the buffer, the directed graph of the target neural network, and the dimension of a single data point input to the target neural network. The operator output includes the size of the input data that can be batch processed and the address allocation of each operator output stored in the cache during the calculation process.
4. The hardware-accelerated scheduling algorithm for neural networks based on directed graph representation according to claim 1, characterized in that: in, The operators include convolution, fully connected, batch normalization, bias processing, vector multiplication, vector addition, and activation functions.