A deep learning compiler based on hardware computation graph

Abstract

This invention discloses a deep learning compiler based on a hardware computation graph, comprising a software computation graph generation module, a hardware computation graph generation module, a software computation graph optimization module, a hardware computation graph optimization module, and a code generation module. The software computation graph generation module transforms a deep learning model defined by an external framework into an internal software computation graph. The hardware computation graph generation module, guided by the software computation graph, first generates a basic set of nodes, then uses a greedy search to adjust the node set until it meets the computational requirements of the entire deep learning model, and finally generates the edges of the hardware computation graph. The code generation module generates the hardware code and simulation code corresponding to the deep learning model. This invention enables efficient and high-performance deployment of models with complex topologies, reducing deployment costs. Furthermore, this invention can be applied to various hardware platforms such as CPUs, GPUs, FPGAs, and dedicated AI chips.

Description

Technical Field

[0001] This invention relates to a deep learning compiler based on a hardware computation graph, belonging to the field of hardware resource utilization technology for deep learning compilers, and realizing the automated deployment of deep learning models. Background Technology

[0002] In recent years, deep learning has been widely applied in many scenarios, such as image recognition, object detection, and semantic segmentation. At the same time, the efficient acceleration of deep learning models has attracted increasing attention from both academia and industry. Therefore, how to efficiently and effectively automate the deployment of models has become particularly important. Deep learning compilers have been proposed for this purpose. In terms of software architecture, the current mainstream approach is to convert the model defined by the external framework into the compiler's internal representation, optimize it based on the resource availability of the target hardware platform, and then generate hardware code.

[0003] However, as the application scenarios of deep learning continue to expand and the community continues to pursue model accuracy, model topologies are becoming increasingly complex. Existing compilers, when faced with complex topologies, can only provide limited support by using more expensive hardware platforms with greater hardware resources, significantly increasing deployment costs. Summary of the Invention

[0004] Purpose of the Invention: Current deep learning compilers perform poorly in terms of hardware resource utilization, and performance can only be improved by using more expensive hardware platforms with greater hardware resources. Furthermore, as the complexity of model topologies continues to increase, these methods cannot guarantee successful deployment. This invention provides a deep learning compiler based on a hardware computation graph, which can deploy models with complex topologies using fewer hardware resources and effectively improve the efficiency of hardware resource utilization.

[0005] Technical solution: A deep learning compiler based on hardware computation graph, including a software computation graph generation module, a hardware computation graph generation module, a software computation graph optimization module, a hardware computation graph optimization module, and a code generation module;

[0006] The software computation graph generation module is used to convert the deep learning model defined by the external framework into an internal software computation graph.

[0007] The software computation graph optimization module is used to optimize the software computation graph;

[0008] The hardware computation graph generation module, guided by the software computation graph, first generates a basic set of nodes, then uses a greedy search to adjust the set of nodes until it meets the computational requirements of the entire deep learning model, and finally generates the edges of the hardware computation graph.

[0009] The hardware computation graph optimization module is used to optimize the hardware computation graph;

[0010] The code generation module is used to generate hardware code and simulation code corresponding to deep learning models.

[0011] In the software computation graph generation module, the node types and edges in the software computation graph are defined according to the hardware implementation efficiency; the model file is parsed and converted; the nodes in the software computation graph are the operation units of a deep learning model, including operators such as padding, convolution, max pooling, average pooling, global pooling, addition, and concatenation; the edges in the software computation graph represent the data transmitted between nodes.

[0012] In the hardware computation graph generation module, the node types in the hardware computation graph correspond one-to-one with the node types in the software computation graph. However, a node in the hardware computation graph represents a processing unit on the hardware platform; that is, nodes and edges represent modules and hardwired connections within the chip, respectively. Furthermore, in actual implementation, a multiplexer and a multiplexer are added to each input and output port of a node. The multiplexer selects the predecessor node of the current hardware node, and the multiplexer selects the successor node of the current hardware node. In the hardware computation graph generation module, the mapping method from software computation graph nodes to hardware computation graph nodes is based on the concept of multi-core processors, enabling a one-to-many mapping between the two. For a convolution node in the software computation graph, if no existing convolution node in the hardware computation graph can meet the computation requirements, the output channel dimension of the weights of the convolution node in the software computation graph is split (OCP), and the weight data of the convolution node is distributed to multiple hardware nodes (nodes in the hardware computation graph) for computation. For a pooling node in the software computation graph, if no existing pooling node in the hardware computation graph can meet the computation requirements, the width dimension of the input data of the pooling node in the software computation graph is split (IWP). Including the case where node splitting is not required (NP), nine connection methods between nodes in the hardware computation graph are proposed, namely:

[0013] NP→NP, the current software computation graph nodes and their predecessor nodes do not need to be split;

[0014] NP→IWP, the current software computation graph nodes need to be segmented in the width dimension of the input data, while the predecessor nodes do not need to be segmented;

[0015] NP→OCP, the predecessor software computation graph nodes do not need to be segmented, but the current software computation graph nodes need to be segmented in the output channel dimension of the weight data;

[0016] IWP→NP, the current software computation graph node does not need to be segmented, but the predecessor software computation graph node needs to be segmented in the width dimension of the input data;

[0017] IWP→IWP, both the current software computation graph node and the predecessor software computation graph node need to be segmented in the width dimension of the input data;

[0018] IWP→OCP: The current software computation graph node is segmented in the output channel dimension of the weight data, while the predecessor software computation graph node needs to be segmented in the width dimension of the input data.

[0019] OCP→NP, the current software computation graph node does not need to be segmented, but the predecessor software computation graph node needs to be segmented in the output channel dimension of the weight data;

[0020] OCP→IWP: The current software computation graph node is segmented in the output channel dimension of the weight data, and the predecessor software computation graph node needs to be segmented in the output channel dimension of the weight data.

[0021] And for OCP→OCP, both the current software computation graph node and the predecessor software computation graph node need to be segmented in the output channel dimension of the weight data.

[0022] Furthermore, these nine connection methods are implemented in two steps. For example, for OCP→IWP, an OCP→IWP backend module is added to the output port of all hardware modules required by the preceding software computation graph node, and an OCP→IWP frontend module is added to the input port of the hardware modules required by the current layer. The other connection methods are implemented by adding frontend or backend modules to the input or output ports of the corresponding hardware modules according to the same mechanism.

[0023] The specific steps for transforming the model defined by the external framework into an internal software computation graph are as follows: input the model defined by the external framework and trained, and parse the operators therein; reorder the nodes based on the depth-first principle.

[0024] The specific steps of the software computation graph optimization module to optimize the software computation graph are as follows: for nodes in the software computation graph that have more than 2 input ports, implement and replace them with multiple nodes with 2 inputs; for nodes that have more than 2 output ports, add a Duplicate operator at their end to broadcast data; for Duplicate operators that have more than 2 output ports, implement and replace them with multiple Duplicate operators with 2 outputs.

[0025] The specific steps for generating the hardware computation graph are as follows: Based on the types of nodes in the software computation graph and the amount of hardware resources used, frequency statistics are performed on the nodes in the software computation graph, and hardware resources (e.g., memory size, computing resources, etc.) are allocated to the nodes according to their frequency. Then, based on the greedy search approach, considering the total resources of the target hardware platform, and guided by the software computation graph, a set of hardware computation graph nodes that can meet the computational requirements of the entire model is generated. Hardware computation graph nodes represent hardware resources allocated to the software computation graph nodes; they are a data structure used to represent hardware resources. Next, based on the node sets of the software and hardware computation graphs, edges are generated in the hardware computation graph to reflect the interconnection relationships between the nodes. Simultaneously with generating the hardware computation graph edges, the software computation graph is divided into multiple subgraphs, which communicate with each other via off-chip memory. Off-chip memory resource allocation is performed on the data between these subgraphs, implemented using dynamic programming. Each node in the hardware computation graph represents a processing unit, but each time the program runs, some processing units need to retrieve data from off-chip memory.

[0026] The specific steps for optimizing the hardware computation graph in the hardware computation graph optimization module are as follows: traverse each edge in the hardware computation graph and trim edges with duplicate functions.

[0027] Beneficial effects: Compared with existing technologies, the hardware computation graph-based deep learning compiler provided by this invention can not only deploy models with complex topologies using fewer hardware resources, but also achieve better performance for models with low topological complexity. Furthermore, this invention can be applied to various hardware platforms such as CPUs, GPUs, FPGAs, and dedicated AI chips. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the process of an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating the generation process of the software computation graph in an embodiment of the present invention.

[0030] Figure 3 This is an optimized flowchart of the software computation graph according to an embodiment of the present invention;

[0031] Figure 4 This is a flowchart of the first stage of generating the hardware computation graph according to an embodiment of the present invention;

[0032] Figure 5 This is a flowchart of the second stage of generating the hardware computation graph according to an embodiment of the present invention;

[0033] Figure 6 This describes the edge generation process of the hardware computation graph in an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0035] When using FPGA as the target hardware platform, the deep learning compiler based on hardware computation graphs includes a hardware computation graph generation module, a software computation graph optimization module, and a code generation module; the workflow is as follows: Software computation graph generation ( Figure 2 ), software computation graph optimization ( Figure 3 Hardware computation graph generation ( Figure 4 Hardware computation graph optimization Figure 5 ) and code generation.

[0036] The software computation graph generation process implemented by the software computation graph optimization module is as follows: (Step 2.0) Initialize the operator index cur in the model defined by the current external deep learning framework to 0. (Step 2.1) Determine whether the compiler supports the type of the cur-th operator. (Step 2.2) If the compiler does not support the current operator, report an error and exit the program. (Step 2.3) Otherwise, create the compiler's internal representation node (i.e., the node of the software computation graph) based on the operator in the model input by the system. (Step 2.4) Increment cur to point to the next operator. (Step 2.5) Repeat steps 2.1-2.4 until all operators in the model have been traversed. (Step 2.6) Under the guidance of the input model, determine the predecessor and successor nodes of each software computation graph node, thereby generating the edges of the software computation graph.

[0037] In the software computation graph optimization module, the optimization process of the software computation graph is as follows: (Step 3.0) Initialize the operator index cur in the current software computation graph to 0; (Step 3.1) Count the number of input edges n and the number of output edges m of the current node; (Step 3.2) When n is less than or equal to 2, do not process; (Step 3.3) When n is greater than 2, split the input and replace it with (n-1) input nodes of size 2; (Step 3.4) When m is less than 2, do not process; (Step 3.5) When m is greater than or equal to 2, insert a Duplicate operator at the end of the current node. The input is the current node, and the output is all the output edges of the current node. At the same time, the original output edges of the current node are deleted. (Step 3.6) Count the number of output edges k of the current Duplicate node. (Step 3.7) When k equals 2, no processing is performed. (Step 3.8) When k is greater than 2, (k-1) Duplicate nodes are used to replace it. (Step 3.9) Increment cur to point to the next node. (Step 3.10) Repeat steps 3.1-3.9 until all nodes in the software computation graph have been traversed. (Step 3.11) Sort all nodes by depth first.

[0038] In the hardware computation graph generation module, the generation process of the hardware computation graph includes basic node generation, node set adjustment, edge generation, and off-chip storage allocation.

[0039] Basic node generation, which allocates hardware resources to nodes based on frequency, follows these steps: (Step 4.0) Initialize the frequency statistics dictionary `dict`, where the key is the node object and the value is the node frequency (i.e., the number of times the node appears). Initialize the current software node (node ​​in the software computation graph) index `cur` to 0. (Step 4.1) Calculate the on-chip memory size `ram_size` required for the `cur`-th software node. (Step 4.2) Generate the key for the software node using the memory size `ram_size` and the type of the current software node. (Step 4.3) Increment the frequency corresponding to the key. (Step 4.4) Set `cur` to point to the next software node. (Step 4.5) Repeat steps 4.1-4.4 until all nodes in the software computation graph have been traversed. (Step 4.6) Obtain the frequency statistics dictionary `dict_freq` by dividing the frequency in the frequency statistics dictionary `dict` by the total number of nodes in the software computation graph. (Step 4.7) Initialize the current key index `cur_k` to 0. (Step 4.8) Obtain the `cur_k`-th key from `dict_freq`. k, (Step 4.9) Multiply the frequency dict_freq[k] corresponding to the current key by the total number of resources of the target hardware platform to obtain the hardware resources allocated to the current key. (Step 4.10) cur_k points to the next key. (Step 4.13) Repeat steps 4.8-4.10 until all keys in dict_freq have been traversed, that is, all basic hardware nodes are obtained.

[0040] The node set adjustment process is as follows: (Step 5.0) Initialize the current hardware computation graph node index cur to 0; (Step 5.1) Generate a set number (maximum 20 to prevent excessive program execution time) of software computation subgraph candidates (i.e., a continuous set of nodes in the software computation graph) starting from the current node; (Step 5.2) Based on the nine node connection methods proposed in this invention, calculate the required hardware module type (e.g., convolution, pooling, etc.), configuration, and quantity for each subgraph candidate under the current hardware computation graph node set; (Step 5.3) Based on the results in Step 5.2, adjust the current hardware computation graph node set... Subtract the set of computational graph nodes from the set of nodes actually needed by the input model to obtain the difference between the two. (Step 5.4) Without exceeding the total number of resources of the target hardware platform, select the longest one from the candidate subnetworks. (Step 5.5) Add cur to the starting point of the next iteration. (Step 5.6) Supplement the existing hardware node set, that is, construct new hardware computational graph nodes according to the hardware nodes required by the subgraph candidates selected in (Step 5.4) and add them to the existing hardware node set. (Step 5.7) Repeat steps 5.1-5.6 until the traversal of the entire software computational graph is completed.

[0041] The edge generation process is as follows: (Step 6.0) Initialize the current subgraph index cur_g in the software computation graph to 0 and the node index cur_n in the current subgraph to 0. (Step 6.1) Find the set of hardware computation graph nodes required by the cur_n node in the cur_g sub-network. (Step 6.2) Initialize the input node index cur_input in the current software node to 0. (Step 6.4) If the current input software computation graph node is also included in the current subgraph, add an edge to the hardware computation graph nodes required by these two software computation graph nodes. (Step 6.5) If the current input software computation graph node is not included in the current subgraph, add an edge between the read port of the FPGA's off-chip storage communication module and the hardware computation graph node required by the current software computation graph node. (Step 6.6) Increment cur_input to point to the next input port. (Step 6.7) Repeat steps 6.3-6. 6. Continue until all input ports of the current software computation graph node have been traversed. (Step 6.8) Initialize the input node index cur_output of the current software computation graph node to 0. (Step 6.10) If the current output software computation graph node is also included in the current subgraph, add an edge to the hardware computation graph node required by these two software computation graph nodes. (Step 6.11) If the current output node is not included in the current subgraph, add an edge between the write port of the off-chip storage communication module and the current node. (Step 6.12) Increment cur_output to point to the next output port. (Step 6.13) Repeat steps 6.9-6.12 until all output ports of the current node have been traversed. (Step 6.14) Increment cur_n to point to the next node in the software computation graph. (Step 6.15) Repeat steps 6.1-6.14 until all nodes in the software computation graph have been traversed.

[0042] In the hardware computation graph optimization module, the optimization process of the hardware computation graph is as follows: (Step 7.0) Traverse all hardware nodes in the hardware computation graph; (Step 7.1) Traverse each output edge of each hardware node; (Step 7.2) Count the edges with the same function among all output edges of the current hardware node; (Step 7.3) Delete the output edges with duplicate functions.

Claims

1. A deep learning compiler based on a hardware computation graph, characterized in that, It includes a software computation graph generation module, a hardware computation graph generation module, a software computation graph optimization module, a hardware computation graph optimization module, and a code generation module; The software computation graph generation module is used to convert the deep learning model defined by the external framework into an internal software computation graph. The software computation graph optimization module is used to optimize the software computation graph; The hardware computation graph generation module, guided by the software computation graph, first generates a basic set of nodes, then uses a greedy search to adjust the set of nodes until it meets the computational requirements of the entire deep learning model, and finally generates the edges of the hardware computation graph. The hardware computation graph optimization module is used to optimize the hardware computation graph; The code generation module is used to generate hardware code and simulation code corresponding to deep learning models; For nodes with more than two input ports in the software computation graph, implement and replace them using multiple nodes with two input ports; for nodes with more than two output ports, add broadcast nodes at the end of the nodes. Replace broadcast nodes with more than two output ports with multiple broadcast nodes that have only two output ports; reorder the nodes based on a depth-first principle.

2. The deep learning compiler based on hardware computation graph according to claim 1, characterized in that, In the software computation graph generation module, the node types and edges in the software computation graph are defined according to the hardware implementation efficiency; the model file is parsed and converted; the nodes in the software computation graph are the computing units of a deep learning model; and the edges in the software computation graph represent the data transmitted between nodes.

3. The deep learning compiler based on hardware computation graph according to claim 1, characterized in that, In the hardware computation graph generation module, the node types in the hardware computation graph correspond one-to-one with the node types in the software computation graph. However, a node in the hardware computation graph represents a processing unit on the hardware platform. That is, the node and the edge represent the module and hardwire inside the chip, respectively. In actual implementation, a multiplexer and a multiplexer are added to each input port and output port of the node, respectively. The multiplexer is used to select the predecessor node of the current hardware node, and the multiplexer is used to select the successor node of the current hardware node.

4. The deep learning compiler based on hardware computation graph according to claim 1, characterized in that: The specific steps for generating the basic node set are as follows: cluster the nodes according to the type of nodes in the software computation graph and the amount of on-chip resources required; prioritize resource allocation for frequently occurring nodes.

5. The deep learning compiler based on hardware computation graph according to claim 1, characterized in that: The specific steps for adjusting the node set using greedy search until the computational requirements of the entire model are as follows: input the software computation graph and the amount of resources of the target hardware platform; use the greedy search strategy to traverse the nodes of the software computation graph and generate some candidate subnetworks. Calculate the number and configuration of modules required for each subnetwork under the existing hardware; calculate the number and configuration of missing modules for each subnetwork under the existing hardware; select the subnetwork with the longest length without exceeding the resource limit of the target hardware platform; repeat the above steps until all nodes have been traversed.

6. The deep learning compiler based on hardware computation graph according to claim 1, characterized in that: The specific steps for optimization on the hardware computation graph are as follows: traverse the edges of each node in the hardware computation graph and prune the edges with the same function in the hardware computation graph.

Images