Processing unit, related device and method

Abstract

The present disclosure provides a processing unit, related devices and methods. The processing unit includes: an instruction fetch unit; an instruction decoding unit; an instruction execution unit, which is used to: extract a constant subgraph from a static graph of a deep learning model, wherein the constant subgraph is a connected subgraph composed of nodes whose output values ​​are constants under the assumption that the input tensor of the deep learning model has a fixed shape; input any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtain the constant output by the output node of the constant subgraph; construct a constant node with the obtained constant, and replace the constant subgraph in the deep learning model with the constant node to form a second deep learning model. The embodiment of the present disclosure further reduces the amount of data transmitted between the processing unit and the acceleration unit when the deep learning model is running, thereby improving the execution efficiency of the deep learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more particularly, to a processing unit, related devices, and methods. Background Art

[0002] Traditional processing units (such as CPUs, central processing units) are often inefficient when performing computations in specialized areas (such as deep learning models). To improve processing efficiency in specialized areas (such as deep learning), specialized accelerators, such as neural network processing units (NPUs), are used. These accelerators perform specialized deep learning processing (such as neural network processing) under the control of the processing unit. Because the accelerators have on-chip memory, the overhead of frequent data transfers between the processing unit and the accelerators during neural network computations is reduced.

[0003] At present, the deployment of deep learning models to acceleration units is mainly based on the static graph of the deep learning model. The static graph is a pre-defined graph that reflects the various computing nodes of the deep learning model and the relationships between the various computing nodes. The model code compiled during actual runtime is completely compiled according to the static graph. Since it is defined first and then run, there is no need to build a new calculation graph each time it is run, and the running speed is faster. In this way, based on the static graph, the deep learning model can be converted (that is, from a specific format or framework to a universal format expression), and quantization nodes are inserted before the specific node positions of the static graph for quantization. The quantized deep learning model is compiled into an acceleration unit model that can be recognized by the acceleration unit and deployed to the acceleration unit for execution.

[0004] In the static graph of a deep learning model, some nodes are constant nodes, whose outputs are always constant. In some areas of the static graph, these constant nodes are interconnected, forming a constant subgraph. The operations on these constant nodes are difficult to accelerate using accelerators dedicated to deep learning model processing. Their presence results in a large number of constants being transferred between the processing and accelerator units during model execution, reducing the execution efficiency of the deep learning model. In practice, since operations between constants are still constants, there's no need to deploy each of these constant nodes to the accelerator unit. Instead, the final constant operation result of this interconnected constant node in that area needs to be deployed as a single node to the accelerator unit. Replacing this interconnected constant node with a single, unified constant result node reduces the amount of data transferred between the processing and accelerator units during actual model execution, improving model execution efficiency. This approach is called constant folding. However, since this technique only folds connected constant nodes, there's still room for improvement in folding efficiency, and model execution efficiency needs to be further improved. Summary of the Invention

[0005] In view of this, the present disclosure aims to further reduce the amount of data transmitted between the processing unit and the acceleration unit when the deep learning model is running, thereby improving the execution efficiency of the deep learning model.

[0006] According to one aspect of the present disclosure, there is provided a processing unit, comprising:

[0007] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0008] an instruction decoding unit for decoding the retrieved computer instructions;

[0009] An instruction execution unit is configured to execute the decoded computer instructions to achieve: extracting a constant subgraph from a static graph of a deep learning model, wherein the constant subgraph is a connected subgraph composed of nodes whose output values ​​are constants, assuming that an input tensor of the deep learning model has a fixed shape; inputting any tensor satisfying the fixed shape into the static graph as an input tensor, and obtaining a constant output by an exit node of the constant subgraph, wherein the exit node refers to a node in the constant subgraph whose output end points to an external node of the constant subgraph and whose input end is an internal node of the constant subgraph; constructing a constant node using the obtained constant, and replacing the constant subgraph in the deep learning model with the constant node to form a second deep learning model.

[0010] Optionally, extracting a constant subgraph from a static graph of the deep learning model includes:

[0011] Marking constant nodes and shape-related nodes that are only related to the shape of the input tensor of the deep learning model in the static graph;

[0012] Based on the marked constant nodes and shape-related nodes, the constant subgraph is extracted.

[0013] Optionally, extracting the constant subgraph based on the marked constant nodes and shape-related nodes includes:

[0014] Repeat the process of marking the nodes in the static graph whose input ends are all marked nodes until all the nodes whose input ends are all marked nodes are marked;

[0015] Divide the interconnected nodes among the marked nodes into a constant subgraph.

[0016] Optionally, inputting any tensor satisfying the fixed shape as an input tensor into the static graph, and obtaining a constant output by an exit node of the constant subgraph, includes:

[0017] Constructing a tensor satisfying the fixed shape;

[0018] Inputting the constructed tensor as an input tensor into the static graph;

[0019] Get the constant output by the exit node of the constant subgraph.

[0020] Optionally, constructing constant nodes using the acquired constants, and replacing the constant subgraph in the deep learning model with the constant nodes to form a second deep learning model includes:

[0021] Deleting the constant subgraph;

[0022] A constant node is constructed using the obtained constant, where the constructed constant node points to the external node pointed to by the output terminal of the exit node of the deleted constant subgraph.

[0023] Optionally, deleting the constant subgraph includes:

[0024] Deleting the pointing relationship between the exit node and the external node pointed to by the output end of the exit node;

[0025] Repeat the process of deleting nodes with an out-degree of 0 in the static graph until no nodes with an out-degree of 0 exist, wherein the out-degree is the number of nodes pointed to by the output end of the node.

[0026] Optionally, the instruction execution unit is further used to execute the decoded computer instructions to achieve: after forming the second deep learning model, converting the second deep learning model into a universal intermediate expression.

[0027] Optionally, the instruction execution unit is also used to execute the decoded computer instructions to achieve: after converting the second deep learning model into a universal intermediate expression, inserting a maximum and minimum value collection node into the universal intermediate expression to collect the maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression with the maximum and minimum value collection node inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression with the quantization node and dequantization node inserted back to the specific format to obtain the quantized model.

[0028] Optionally, the instruction execution unit is also used to execute the decoded computer instructions to achieve: after obtaining the quantized model, converting the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit, and sending it to the acceleration unit.

[0029] According to one aspect of the present disclosure, there is provided a computing device, comprising:

[0030] a processing unit as described above;

[0031] a memory for storing the computer instructions;

[0032] The acceleration unit is used to run the deep learning model.

[0033] According to one aspect of the present disclosure, a system on chip is provided, comprising an acceleration unit for running the deep learning model, wherein the deep learning model is processed by the processing unit as described above and then deployed on the acceleration unit.

[0034] According to one aspect of the present disclosure, a data center is provided, comprising the computing device described above.

[0035] According to one aspect of the present disclosure, a processing method is provided, comprising:

[0036] Extracting a constant subgraph from a static graph of a deep learning model, wherein the constant subgraph is a connected subgraph consisting of nodes whose output values ​​are constants assuming that an input tensor of the deep learning model has a fixed shape;

[0037] Input any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtain a constant output by an exit node of the constant subgraph, where the exit node refers to a node in the constant subgraph whose output terminal points to an external node of the constant subgraph and whose input terminal is an internal node of the constant subgraph;

[0038] A constant node is constructed using the obtained constants, and the constant subgraph in the deep learning model is replaced by the constant node to form a second deep learning model.

[0039] Optionally, extracting a constant subgraph from a static graph of the deep learning model includes:

[0040] Marking constant nodes and shape-related nodes that are only related to the shape of the input tensor of the deep learning model in the static graph;

[0041] Based on the marked constant nodes and shape-related nodes, the constant subgraph is extracted.

[0042] Optionally, extracting the constant subgraph based on the marked constant nodes and shape-related nodes includes:

[0043] Repeat the process of marking the nodes in the static graph whose input ends are all marked nodes until all the nodes whose input ends are all marked nodes are marked;

[0044] Divide the interconnected nodes among the marked nodes into a constant subgraph.

[0045] Optionally, inputting any tensor satisfying the fixed shape as an input tensor into the static graph, and obtaining a constant output by an exit node of the constant subgraph, includes:

[0046] Constructing a tensor satisfying the fixed shape;

[0047] Inputting the constructed tensor as an input tensor into the static graph;

[0048] Get the constant output by the exit node of the constant subgraph.

[0049] Optionally, constructing constant nodes using the acquired constants, and replacing the constant subgraph in the deep learning model with the constant nodes to form a second deep learning model includes:

[0050] Deleting the constant subgraph;

[0051] A constant node is constructed using the obtained constant, where the constructed constant node points to the external node pointed to by the output terminal of the exit node of the deleted constant subgraph.

[0052] Optionally, deleting the constant subgraph includes:

[0053] Deleting the pointing relationship between the exit node and the external node pointed to by the output end of the exit node;

[0054] Repeat the process of deleting nodes with an out-degree of 0 in the static graph until no nodes with an out-degree of 0 exist, wherein the out-degree is the number of nodes pointed to by the output end of the node.

[0055] In a deep learning model, in addition to the constant nodes whose outputs are always constant, there are also a large number of non-constant nodes, but they are nodes whose outputs are only related to the shape of the input tensor of the deep learning model. Since in most cases, the input tensors of the deep learning model are of fixed shape, it can be considered that the outputs of these nodes are also fixed in most cases. When determining the constant subgraph in the static graph, the embodiment of the present disclosure not only considers the constant nodes, but also takes into account the nodes that are only related to the shape of the input tensor of the deep learning model, and also takes into account the influence of these two types of nodes on the downstream nodes, and also takes into account the nodes whose outputs become fixed due to the influence of these two types of nodes. Based on all these nodes with fixed outputs, a constant subgraph is determined together, and a constant node is constructed with the constant output constant of the constant subgraph to replace the constant subgraph, further reducing the amount of data transmitted between the processing unit and the acceleration unit when the deep learning model is running, and improving the execution efficiency of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0057] Figure 1 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0058] Figure 2 is an internal structure diagram of a server in a data center according to an embodiment of the present disclosure;

[0059] Figure 3 This is an internal structural diagram of a processing unit and an acceleration unit within a server according to an embodiment of the present disclosure;

[0060] Figure 4 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present disclosure;

[0061] Figure 5 This is an exemplary hierarchical architecture diagram for a deep learning model running platform;

[0062] Figure 6 is a flowchart of the entire process of deploying a deep learning model to an acceleration unit in an embodiment of the present disclosure;

[0063] Figure 7 is an example diagram of performing constant folding on a deep learning model according to one embodiment of the present disclosure;

[0064] Figure 8 The present invention is a flowchart of a method for performing constant folding by a processing unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0066] The following terms are used in this article.

[0067] Deep Learning Model: Deep learning is a new research direction in the field of machine learning (ML). It was introduced to bring ML closer to its original goal: artificial intelligence (AI). Deep learning learns the inherent patterns and representational hierarchies of sample data. The information gained during this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have human-like analytical learning capabilities and the ability to recognize data such as text, images, and sound. The model used in deep learning is called a deep learning model. Deep learning models have different formats depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet.

[0068] Acceleration unit: A processing unit designed to improve the data processing speed in some special-purpose fields (for example, processing images, processing various operations of deep learning models, etc.) where traditional processing units are inefficient. The acceleration unit is also called an artificial intelligence (AI) processing unit, and includes a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated intelligent acceleration hardware (for example, a neural network processor NPU). The present disclosure is applicable to NPU scenarios, but the embodiments of the present disclosure adopt a general-purpose compiled custom interface, and the acceleration unit hardware implementation can adopt the above-mentioned CPU, GPU, GPGPU, etc.

[0069] Processing unit: A unit within a data center server that performs traditional processing (not for complex computations like image processing and deep learning models). In addition to performing these non-image processing and deep learning model-related complex computations, the processing unit also schedules the acceleration units, assigning them tasks. Processing units can take various forms, including CPUs, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0070] Dynamic graph: Currently, deep learning model frameworks are divided into static graph frameworks and dynamic graph frameworks. Deep learning models consist of multiple layers, each with multiple operation nodes. Operation nodes are the basic operation units in deep learning models that perform operations based on input to obtain outputs passed to the next layer, such as Figure 7The convolution 602, the extraction of input tensor shape 603, etc. in each layer are each an operation node. The results output by the operation nodes of each layer are transmitted to the operation nodes of the next layer as inputs received by the operation nodes of the next layer. The graph of a deep learning model refers to a graphical representation of the operation nodes in the deep learning model and the relationships between the operation nodes, which is divided into dynamic graphs and static graphs. A dynamic graph refers to a graph that changes with the actual order of commands written. This means that the commands written each time are different, and the resulting dynamic graphs are different. Based on the commands written each time, the required operation nodes and the execution relationships between the operation nodes are determined, thereby forming different dynamic graphs. This mechanism makes debugging easier and makes it easier to convert ideas in the brain into actual code. This is because the dynamic graph is formed based on the code after the code is written, and there is no need to be restricted by the dynamic graph when writing the code. In the PyTorch neural network framework, dynamic graphs are used.

[0071] Static graph: It is a graph that is pre-generated before writing code based on the structure of the deep learning model and represents the relationship between each operation node in the deep learning model. In other words, the deep learning model is first defined through a static graph. There is no need to rebuild the graph each time the deep learning model is run, so the deep learning model runs faster when using a static graph than a dynamic graph. In theory, a mechanism like a static graph allows the compiler to perform greater optimizations, but this also means that there is more of a generation gap between the expected program and the actual execution of the compiler. This also means that errors in the code will be more difficult to find (for example, if there is a problem with the structure of the static graph, you may only find it when the code executes the corresponding operation).

[0072] Constant subgraph: It is a subgraph composed of several connected nodes in the static graph. These connected nodes are nodes whose output values ​​are constant under the assumption that the input tensor of the deep learning model has a fixed shape. There are several cases for these nodes: the first case is that the node itself is a constant node, such as Figure 7 The second case is that the node itself is not a constant node, but a node that is only related to the shape of the input tensor of the deep learning model. Since the shape of the input tensor of the deep learning model is fixed in most cases, the output of the node is also fixed in most cases, such as Figure 7 The node 603 of the extracted input tensor shape; the third case is a node that does not belong to the first case node or the second case node, but due to the first case node and the second case node, its output is fixed in most cases, such as Figure 7The sharding node 605 has an input that comes from the tensor shape extracted by the Extract Input Tensor Shape Node 603, and the tensor shape extracted by the Extract Input Tensor Shape Node 603 is fixed in most cases, resulting in the output of the sharding node 605 being fixed in most cases. In addition, Figure 7 The input of the sum node 607 comes from the shard node 605 and the constant node 606. Since the output of the shard node 605 and the output of the constant node 606 are fixed in most cases, the output of the sum node 607 is also fixed in most cases. Figure 7 In the example, the input tensor shape extraction node 603, the sharding node 605, the constant node 606 and the node 607 are connected to form a constant subgraph.

[0073] Input tensor: A tensor is a set of ordered numbers that satisfy certain coordinate transformation relationships when the coordinate system is changed. In layman's terms, it is a generalization of vectors and matrices. A scalar is considered a 0th-order tensor, a vector is considered a 1st-order tensor, and a matrix is ​​considered a 2nd-order tensor. However, when the horizontal and vertical dimensions are insufficient to represent the input quantity, tensors of 3rd and higher order arise. Tensors allow us to represent input quantities in any dimensional space. A hallmark of deep learning models is their ability to accept input quantities in any dimensional space. Regardless of the dimensionality of the input, it can be represented as an input tensor, which is fed into the nodes of the first layer of the deep learning model. The nodes in this first layer have weight tensors in the same dimensional space. Because the dimensional space of the input tensor and the weight tensor are the same, operations such as dot products and convolutions can be performed on the input and weight tensors in the same dimensional space, and the resulting output is still in the same dimensional space. The output tensor of the nodes in the previous layer is input to the nodes in the next layer as input, and tensor operations such as dot multiplication and convolution are performed in the same dimensional space with the weight tensor of the nodes in the next layer. This process is repeated until the output tensor of the nodes in the last layer is obtained as the output tensor of the entire deep learning model.

[0074] The shape of the input tensor: It refers to the number of dimensions in each dimensional space of the input tensor. As mentioned above, the input tensor has an indefinite dimensional space, for example, a vector belongs to a one-dimensional space and a matrix belongs to a two-dimensional space. In each dimensional space, the dimension of the input tensor is different. For example, for a 4×5 matrix, it has 2 dimensional spaces. In the first dimensional space, it has 4 dimensions. In the second dimensional space, it has 5 dimensions. Therefore, the shape of the 4×5 matrix input tensor is (4, 5). If an input tensor has 3 dimensional spaces, where there are 6 dimensions in the first dimensional space, 2 dimensions in the second dimensional space, and 8 dimensions in the third dimensional space, then the shape of this input tensor is (6, 2, 8). It has nothing to do with the specific values ​​of each dimension in each dimensional space of the input tensor.

[0075] Connected subgraph: It is a part of the graph. If for any node in this part, at least one other node can be found in this part that has an input-output relationship with it (the input or output of at least one other node is this node), this part is a connected subgraph of the graph.

[0076] Exit node: A node in the constant subgraph whose output end points to an external node of the constant subgraph and whose input end is an internal node of the constant subgraph. Since the output values ​​of the nodes of the constant subgraph are all constants under the premise that the input tensor of the deep learning model has a fixed shape, the exit node is a node that finally outputs the calculation results of these nodes in the constant subgraph to the outside of the constant subgraph. Its input comes from the constants output by the internal nodes of the constant subgraph, so the output is also a constant. This constant is the calculation result of the constants output by the internal nodes of the constant subgraph, and it is output to the nodes outside the constant subgraph as the output of the constant subgraph. Figure 7 The sum node 607 has its input ends as the two constant subgraph internal nodes, the fragmentation node 605 and the constant node 606, and its output end as the re-synthesized shape node 608, which is a constant subgraph external node.

[0077] Constant node: A node that sets a constant value. This node itself does not have any computational function, but the node to which its output terminal points uses the constant value for computation. Figure 7 The constant node 606 does not provide any operation by itself, but the constant set by the constant node 606 and the value generated by the shard node 605 are ANDed in the AND node 607.

[0078] Constant folding: The process of replacing a constant subgraph with constructed constant nodes. Because each node in the constant subgraph outputs a constant value, assuming the input tensor of the deep learning model has a fixed shape, when deploying the accelerator model, it is not necessary to deploy these constant nodes one by one to the accelerator. Instead, the final constant operation result of this interconnected constant node in the region is deployed as a single node to the accelerator. Replacing a network of interconnected constant nodes with a single, unified result constant node reduces the amount of data transmitted between the processing unit and the accelerator during actual model execution, improving model execution efficiency.

[0079] Shape-dependent nodes: Nodes whose output values ​​are only related to the shape of the input tensor of the deep learning model. That is, when the shape of the input tensor of the deep learning model is fixed, the output value of the shape-dependent node is also fixed, such as Figure 7 Extract input tensor shape of node 603.

[0080] Directed relationship: A relationship in a static graph represented by a directed arrow from one node to another, where the output of the node emitting the arrow is sent as input to the node pointed by the arrow.

[0081] Out-degree: The out-degree is the number of nodes that a node's output terminal points to. In a static graph, if a node's output terminal has no pointing relationship to any node, the node's out-degree is 0; if a node's output terminal has an arrow pointing to another node, the node's out-degree is 1; if a node's output terminal has two arrows pointing to two other nodes, the node's out-degree is 2, and so on.

[0082] Quantization: The act of converting the input tensors of a deep learning model, as well as the weight tensors and other parameters of a node, from high-precision data types to low-precision data types, thereby reducing the requirements for data throughput and storage space. For example, quantizing a 32-bit floating-point number to an 8-bit integer. Although this reduces the accuracy of the model, it greatly improves the execution efficiency of the model and reduces the requirements for data throughput and storage space.

[0083] Dequantization: The reverse process of quantization, converting the input tensors of a deep learning model, as well as the weight tensors and other parameters of a node, from low-precision data types to high-precision data types. Dequantization is the reverse of quantization.

[0084] Conversion: Deep learning models have different formats, depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet. The code expressions of these deep learning models also differ. This creates difficulties in subsequent quantization and compilation of deep learning models. Therefore, deep learning models with different model frameworks or formats must be converted into a unified framework or format. This process is called conversion. It analyzes the meaning of each code statement in the deep learning model and translates it into a common expression based on the meaning of the code statement. This ensures that code statements with the same meaning across different deep learning models are expressed identically in this common expression. Currently, there are tools that convert the expressions of different deep learning models into a common expression.

[0085] Common intermediate expression: The above conversion process transforms deep learning models of different model frameworks or formats into the above common expression form.

[0086] Calibration: As in the quantization example above, quantization first requires determining the value range of the inputs of the operation nodes in the deep learning model. Only by determining the maximum and minimum values ​​of the input to be quantized can the maximum and minimum values ​​of the input to be quantized be mapped to the maximum and minimum values ​​after quantization, thus completing the quantization of the input to be quantized. Calibration is the process of determining the maximum and minimum values ​​of the input to be quantized.

[0087] A Maximum / Minimum Value Collection node is a program that is placed at a specific location in a program and is used to collect the maximum and minimum values ​​generated at that location during program execution. It includes the program code that collects the values ​​generated at that location and determines the maximum and minimum values, as well as the data required for this process.

[0088] Quantization and Dequantization nodes are placed at specific locations in the program to perform the aforementioned quantization and dequantization operations, respectively, and their associated data. This way, when a deep learning model runs on a quantization node, quantization is performed; when it runs on a dequantization node, dequantization is performed.

[0089] Quantized model: The model generated by the above deep learning model after quantization.

[0090] Compilation: Deep learning models are only supported by the instruction set of the processing unit, not the instruction set of the acceleration unit, and therefore cannot be directly executed on the acceleration unit. In order to enable the deep learning model to be supported by the instruction set of the acceleration unit and executed on the acceleration unit, compilation is required. That is, the deep learning model is converted into a model that can be supported by the instruction set of the acceleration unit.

[0091] Acceleration unit model: The deep learning model is converted through the above compilation process into a model that can be supported by the instruction set of the acceleration unit.

[0092] Application environment of the present disclosure

[0093] The embodiments of the present disclosure propose a universal quantization scheme suitable for various deep learning models. The entire compilation optimization scheme is relatively universal. The model after quantization is completed on the processing unit can be used for various hardware devices that execute the model, for example, for data centers, for AI (artificial intelligence) acceleration units, for GPUs (graphics processing units), for IOT (Internet of Things) devices that can execute deep learning models, embedded devices, etc. This compilation optimization method is independent of the hardware on which the model is ultimately deployed. However, for the sake of exemplary description, the following description will mainly focus on the data center as the application scenario. Those skilled in the art should understand that the embodiments of the present disclosure can also be applied to other application scenarios.

[0094] Data Center

[0095] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's network infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Deep learning, a key AI technology, has been widely applied to big data analytics in data centers.

[0096] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in Figure 1, this is the hierarchical inter-networking model. This model consists of the following parts:

[0097] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0098] Access switch 130: Access switches 130 connect servers 140 to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, and are therefore also called top-of-rack switches. They physically connect to the servers.

[0099] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0100] Core switch 110: Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for aggregation switch 120. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0101] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0102] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with other aggregation switches 120 being used only in the event of a failure (dashed lines in the figure above). This means that horizontal scalability is impossible at the aggregation switch 120 level because even if multiple aggregation switches 120 are added, only one is still operational.

[0103] server

[0104] Since the server 140 is the real processing device in the data center, Figure 2 The figure shows a block diagram of the internal structure of a server 140. The server 140 includes a memory 210, a processing unit cluster 270 and an acceleration unit cluster 280 connected by a bus. The processing unit cluster 270 includes multiple processing units 220. The acceleration unit cluster 280 includes multiple acceleration units 230. The acceleration unit 230 is a processing unit designed to improve the data processing speed in a specific application field. The acceleration unit is also called an artificial intelligence (AI) processing unit, which includes a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated intelligent acceleration hardware (for example, a neural network processor NPU). The embodiment of the present disclosure is applicable to the NPU scenario, but due to the use of a general compiled custom interface, CPU, GPU, GPGPU, etc. can also be used in hardware. The processing unit is a processing unit that schedules the acceleration unit and assigns the sequence of instructions to be executed to each acceleration unit. It can take various forms such as a processing unit (CPU), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0105] The architecture design of the traditional processing unit makes the control unit and storage unit occupy a large part of the space in the architecture, while the space occupied by the computing unit is insufficient. Therefore, it is very effective in terms of logical control, but not efficient enough in terms of large-scale parallel computing. Therefore, various special acceleration units have been developed to more effectively improve the computing speed of calculations for different functions and different fields. The acceleration unit proposed in the present invention is a processing unit dedicated to accelerating the computing speed of deep learning models. It is a processing unit that uses a data-driven parallel computing architecture to process a large number of operations (such as convolution, pooling, etc.) of each deep learning model node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each deep learning model node are closely related in the entire computing process and will be used frequently, using the existing processing unit architecture, since the memory capacity in the core of the processing unit is very small, a large number of external core memories must be frequently accessed, resulting in inefficient processing. By using this acceleration unit specifically designed to accelerate the computing processing speed of deep learning models, each core of the unit has on-chip memory with a storage capacity suitable for deep learning model calculations, thereby avoiding frequent access to memory outside the core, it can greatly improve processing efficiency and computing performance.

[0106] The acceleration unit 230 must accept the scheduling of the processing unit 220. Figure 2 As shown, the memory 210 stores various deep learning models, including the nodes of these models and the weight tensors of the nodes. These deep learning models are used when needed. Figure 2 A processing unit 220 is deployed to an acceleration unit 230. That is, the processing unit 220 can send the address of the parameters in the model (such as the weight tensor of each node) in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually uses the deep learning model for calculation, it will directly address these parameters in the memory 210 according to the address of these parameters (such as weight tensors) in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the deep learning model for calculation, the processing unit 220 will also send the input tensor of the model to the acceleration unit 230 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these input tensors and the parameters in the model (such as weight tensors).

[0107] Internal structure of the processing unit and acceleration unit

[0108] The following combination Figure 3 The internal structure diagram of the processing unit 220 and the acceleration unit 230 specifically illustrates how the processing unit 220 schedules the acceleration unit 230 to work.

[0109] like Figure 3As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0110] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 3 The register file 229 shown is used to store a register for instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0111] After fetching the instruction, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0112] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0113] After the instruction issuing unit 225 sends the instruction to the instruction executing unit 226, the instruction executing unit 226 begins executing the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is an instruction for deep learning model inference, the instruction executing unit 226 will no longer execute the instruction. Instead, it will send the instruction to the acceleration unit 230 via the bus for execution.

[0114] Although the disclosed embodiment is used in NPU scenarios, due to the use of a universal compiled custom interface, Figure 3The acceleration unit 230 shown is not limited to NPU, but can also be GPGPU. GPGPU is a general-purpose graphics processor, which is a graphics processor that uses graphics processing tasks to calculate general-purpose computing tasks originally handled by the central processing unit. These general-purpose calculations often have nothing to do with graphics processing. Due to the powerful parallel processing capabilities and programmable pipelines of modern graphics processors, stream processors can process non-graphic data. Especially when facing single instruction stream multiple data streams (SIMD), and the amount of data processing operations is far greater than the needs of data scheduling and transmission, the general-purpose graphics processor greatly surpasses the traditional central processing unit application in performance, so it can play an accelerating role and belongs to the acceleration unit 30 like the NPU. Even if calculations are performed in the NPU scenario, it can still play a role. In addition, the acceleration unit 30 can also be a CPU, GPU, FPGA, ASIC, etc.

[0115] The acceleration unit 230 includes multiple cores 236 ( Figure 3 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0116] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0117] Direct Memory Access (DMA) mechanism 235 is a feature provided by some computer bus architectures that enables data to be written directly from an attached device to the computer's mainboard memory. This greatly improves data access efficiency compared to a method where all data transfers between devices must pass through the processing unit. Because of this mechanism, the cores of the acceleration unit 230 can directly access memory 210 and read parameters in the deep learning model (such as the weight tensor of each node), greatly improving data access efficiency.

[0118] The command processor 237 assigns instructions sent from the processing unit 220 to the acceleration unit 230 to be executed by the core 236. The instruction execution unit 226 sends the acceleration unit 230 a sequence of pending instructions to be executed. After entering the bus channel 231, the pending instruction sequence is cached in the command processor 237, which then selects a core 236 and assigns the instruction sequence to it for execution. The command processor 237 also manages synchronization between the cores 236.

[0119] Acceleration unit core

[0120] Figure 4 FIG. 4 is a diagram of the internal structure of an acceleration unit core according to an embodiment of the present disclosure.

[0121] In one embodiment, Figure 4 As shown, the acceleration unit core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0122] The instruction sequence assigned by the command processor 237 to the acceleration unit core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and assigns them to the tensor engine 310 or the pooling engine 320 for execution based on the nature of the instruction. The tensor engine 310 is responsible for processing operations such as convolution and matrix multiplication in deep learning models. The pooling engine 320 is responsible for processing pooling operations in deep learning models. The memory copy engine 330 is a unit specifically responsible for data copying. This data copying involves copying some data from the on-chip memory 360 to the shared memory of each core 236, or to the on-chip memory 360 of other cores 236, due to potential overflow of the on-chip memory 360. The sequencer 350 determines whether to assign the instruction to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 based on the nature of the operation, such as convolution, matrix multiplication, pooling, or data copy.

[0123] The on-chip memory 360 is an in-core memory that stores the weight tensors in the deep learning model, as well as the input tensors and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning model (for example, hyperparameters in the neural network model) in addition to the weight tensors. As described above, in the process of the processing unit 220 pre-configuring the deep learning model in the acceleration unit 230, the processing unit 220 sends the address of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weight tensors and other parameters (such as hyperparameters) of the nodes. For the weight tensor, the acceleration unit 230 takes it out from the corresponding position of the memory 210 during the actual deep learning model operation and places it in the on-chip memory 360. For other parameters, the acceleration unit 230 takes it out from the corresponding position of the memory 210 during the actual deep learning model operation and places it in the constant buffer 370. In addition, when the instruction to actually start inference is assigned by the command processor 237 to the core 236 for execution, the input tensors in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0124] Software architecture for running deep learning models

[0125] Figure 1-4 The hardware structure that the deep learning model relies on is described from large to small. Figure 5 Describes the software architecture that deep learning models rely on, that is, the hierarchical architecture of the deep learning model operation platform. Figure 5 As shown, from top to bottom, the architecture diagram includes an application layer 401 , a framework layer 402 , and a function layer 403 .

[0126] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, and recommendation 407. These applications are built using this architecture and can also call the architecture to provide runtime interfaces to gain reasoning capabilities within the application.

[0127] The framework layer 402 integrates open source frameworks such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to enable continued optimization and improvement of various algorithms. In each framework, the expression format of deep learning models is different. TensorFlow 408 is a symbolic mathematical system based on data flow programming, which is widely used in the programming implementation of various machine learning algorithms. MXNet 409 is the deep learning library selected by Amazon. Caffe 410, the full name of which is Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modular thinking.

[0128] The functional layer 403 includes a compilation stack 403 and a runtime stack 404. The compilation stack 403 is used to perform conversion (converter) 411, quantization (quantization) 412, optimization (optimization) 413 and compilation (compile) 414 on various models. Conversion 411 is to provide the internal data of the model into a common intermediate expression (IR) format. That is, the framework layer 420 has multiple frameworks such as TensorFlow408, MXNet409, Caffe 410, and different frameworks have different formats. Conversion 411 converts different formats into a unified intermediate expression format. Quantization 412 is to convert the weight tensors in the deep learning model and the input tensors to the deep learning model from high-precision data types to low-precision data types. Optimization 413 is to fuse the operators inside the model, optimize the links of multiple models, and other operations. Compilation 414 is to convert the deep learning variation model according to the instruction set that the acceleration unit can support, and convert it into an acceleration unit model that the acceleration unit can recognize. Constant folding 409 is an embodiment of the present disclosure in Figure 5The added module is an improvement to the disclosed embodiment. It replaces the constant subgraph in the static graph of the deep learning model with constructed constant nodes. Since each node in the constant subgraph outputs a constant value, assuming the input tensor of the deep learning model has a fixed shape, when deploying the acceleration unit model, it is not actually necessary to deploy these constant nodes one by one to the acceleration unit. Instead, the final constant operation result of this interconnected constant node in the region is deployed as a node to the acceleration unit, i.e., constant folding. The runtime stack 404 includes a runtime API 415, an execution manager 416, a user-mode driver 417, and a kernel-mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized runtime API 415 provides various runtime callable interfaces. The user-mode driver 417 provides kernel-mode hardware commands and resource scheduling. The kernel-mode driver 418 provides kernel-mode task scheduling and hardware control.

[0129] The entire process of running the deep learning model in this disclosure embodiment

[0130] Figure 5 The compilation stack 403 includes functions such as conversion 411, quantization 412, optimization 413, compilation 414, and constant folding 409. The actual execution order of these functions is as follows: Figure 6 shown. Figure 6 The process of generating an acceleration unit model that can be recognized by the acceleration unit 230 is shown.

[0131] The deep learning model first undergoes constant folding 409 of the embodiment of the present disclosure, replacing the constant subgraph in its static graph with the constructed constant nodes, thereby greatly reducing the amount of data transmitted between the processing unit 220 and the acceleration unit 230 during the actual operation of the model. The specific process is described below. Figure 8 Detailed description.

[0132] Then, the deep learning model after constant folding 409 is converted 411, that is, converted from various framework formats such as TensorFlow 408, MXNet 409, Caffe 410 into a common intermediate representation.

[0133] Then, the converted deep learning model is quantized 412, and the weight tensors in the deep learning model and the input tensors to the deep learning model are converted from high-precision data types to low-precision data types, thereby reducing the storage space required by the model and improving the model operation efficiency. The specific process of quantization is as follows: inserting a maximum and minimum value collection node (described in detail in the above terminology explanation) into the universal intermediate expression converted by the above conversion process 411 to collect the maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression with the inserted maximum and minimum value collection node back to the specific format before conversion 411 (such as TensorFlow 408, etc.) for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes (described in detail in the above terminology explanation) into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression with the inserted quantization node and dequantization node back to the specific format before conversion 411 to obtain the quantized model.

[0134] Finally, the quantized deep learning model is compiled 414 into an acceleration unit model that can be recognized by the acceleration unit 230. That is, the processing unit 220 converts the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit 230 and sends it to the acceleration unit 230.

[0135] It should be noted that the above-mentioned constant folding 409, conversion 411, quantization 412, and compilation 414 processes are all performed in the processing unit 220. The above process generates an acceleration unit model. After the processing unit 220 deploys the acceleration unit model in the acceleration unit 230, the actual inference process is performed by the acceleration unit 230. That is, the acceleration unit 230 loads the weight tensor in the acceleration unit model into the on-chip memory 360 of the core 236, and also loads the input tensor in the received inference instruction into the on-chip memory 360 of the core 236. The core 236 performs operations based on the weight tensor and the input tensor, and finally obtains the running result of the deep learning model.

[0136] Specific process of constant folding of deep learning model in the embodiment of the present disclosure

[0137] Deep learning models have a static graph. A static graph is a predefined graph that reflects the various computational nodes of a deep learning model and the relationships between them. There is a one-to-one correspondence between the static graph and the deep learning model. When determining whether deep learning model execution can be simplified, the static graph is often considered in conjunction with it. In a deep learning model's static graph, some nodes are constant nodes, whose outputs are always constant. In certain areas of the static graph, these constant nodes are interconnected, forming a constant subgraph. Operations on these constant nodes are difficult to accelerate using accelerators dedicated to deep learning model processing. Their presence causes a large number of constants to be transferred between the processing unit and the accelerator during model execution, reducing the execution efficiency of the deep learning model. In practice, since operations between constants are still constants, simply deploying the final constant operation results of this interconnected region of constant nodes as a single node to the accelerator can ensure the correct operation of the deep learning model while reducing the amount of data transferred between the processing unit and the accelerator during actual model execution, thereby improving model execution efficiency. The existing technology only folds connected constant nodes, and the folding efficiency still has room for optimization, and the model execution efficiency needs to be further improved.

[0138] The inventors of the present disclosure have discovered that there are a large number of nodes in the deep learning model that are only related to the shape of the input tensor of the deep learning model, such as Figure 7 Extract input tensor shape node 603 in. The shape of the input tensor refers to the number of dimensions in each dimensional space of the input tensor. The input tensor has an indefinite dimensional space, for example, a vector belongs to a one-dimensional dimensional space, and a matrix belongs to a two-dimensional dimensional space. In each dimensional space, the dimension of the input tensor is different. For example, for a 4×5 matrix, it has 2 dimensional spaces. In the first dimensional space, it has 4 dimensions. In the second dimensional space, it has 5 dimensions. Therefore, the shape of the input tensor of the 4×5 matrix is ​​(4, 5). If an input tensor has 3 dimensional spaces, where there are 6 dimensions in the first dimensional space, 2 dimensions in the second dimensional space, and 8 dimensions in the third dimensional space, the shape of the input tensor is (6, 2, 8). The shape of the input tensor of the deep learning model is fixed in most cases, so in addition to the constant nodes, the outputs of these nodes that are only related to the shape of the input tensor of the deep learning model are also fixed in most cases. If these nodes are taken into account when extracting the constant subgraph, the coverage of the connected subgraph will be greatly increased, the folding efficiency will be further optimized, and the model execution efficiency will be improved.

[0139] In the embodiment of the present disclosure, the processing unit 220 first marks the constant nodes and the shape-related nodes that are only related to the shape of the input tensor of the deep learning model in the static graph of the deep learning model. The constant node is a node that is a constant value itself. The node itself does not have a calculation function, and the node pointed to by its output terminal uses the constant value for calculation. Figure 7 The constant node 606 does not have any operation function itself, but the constant set by the constant node 606 and the value generated by the slice node 605 are ANDed in the AND node 607. Shape-related nodes are nodes that are only related to the shape of the input tensor of the deep learning model, such as Figure 7 Extract Input Tensor Shape node 603 in.

[0140] After marking the constant nodes and shape-associated nodes, in fact, the output values ​​of these two types of nodes are not the only constants in the static graph. Among the downstream nodes of these two types of nodes (the nodes to which the output terminals of these two types of nodes point), some downstream nodes receive the outputs of these two types of nodes at their input terminals. In this way, the constant values ​​obtained by performing operations based on the constant values ​​are still constant values. Therefore, such nodes can also be divided into the constant subgraph. However, if there are multiple input terminals of the downstream node, and one of them is not a constant input terminal, it cannot be considered that its output value is necessarily constant and cannot be included in the constant subgraph. Therefore, after marking the constant nodes and shape-associated nodes, the nodes whose input terminals are all marked nodes in the static graph are marked (the inputs of these nodes are all constants, and the outputs are also necessarily constants), and this process is repeated continuously (so that the nodes that receive the constant outputs of these nodes and perform further operations to output constants are also marked) until all nodes whose input terminals are all marked nodes are marked. Then, the interconnected nodes among the marked nodes are divided into a constant subgraph. Connectivity means that for any of these nodes, at least one other node can be found in these nodes to have an input-output relationship with it (the input or output of at least one other node is the node). Figure 7 As shown, the only input of the sharding node 605 comes from the output of the extract input tensor shape node 603, and the output value of the extract input tensor shape node 603 is a constant under the premise that the input tensor of the deep learning model has a fixed shape. In this way, under the same premise, the value output by the output of the sharding node 605 is also a constant. In addition, the input of the sum node 607 comes from the output of the sharding node 605 and the output of the constant node 606, and these two nodes output a constant value under the premise that the input tensor of the deep learning model has a fixed shape. In this way, under the same premise, the value output by the output of the sum node 606 is also a constant. Mark these nodes. The marked extract input tensor shape node 603, sharding node 605, constant node 606, and sum node 607 are connected, so they form a connected subgraph, as shown in Figure 7 The dashed ellipse is shown.

[0141] Since the output of the shape-related node is only related to the shape of the input tensor of the deep learning model and has nothing to do with the specific dimensional values ​​in each dimensional space of the input tensor, a tensor that satisfies the fixed shape can be arbitrarily constructed and input into the static graph as the input tensor. The constants output by the exit node of the constant subgraph should all be the same. For example, for a 4×5 matrix, the Extract Input Tensor Shape node 603 simply extracts the shape of the matrix, i.e., (4, 5), indicating that there are 4 elements in each row and 5 elements in each column. Regardless of the 4×5=20 elements, the Extract Input Tensor Shape node 603 outputs (4, 5). The slicing node 605 only outputs 4, plus the value output by the constant node 606, as the exit node, i.e., the value output by the sum node 606. This value is completely unrelated to the specific values ​​of the 4×5=20 elements in the input matrix. Therefore, using any tensor that satisfies the fixed shape, the constant output by the exit node can be obtained.

[0142] An export node is a node in a constant subgraph whose output terminal points to an external node of the constant subgraph and whose input terminal is an internal node of the constant subgraph. Figure 7 The sum node 607 has two internal constant subgraph nodes, the slice node 605 and the constant node 606, as inputs, and one external constant subgraph node, the resynthesized shape node 608, as output. Therefore, it is an exit node. A constant subgraph can have multiple exit nodes.

[0143] Then, the constant subgraph can be deleted, and a constant node can be constructed based on the obtained constants. The constructed constant node points to the external node pointed to by the output end of the export node of the deleted constant subgraph, thereby replacing the constant subgraph.

[0144] When deleting the constant subgraph, the pointing relationship between the exit node and the external node pointed to by the output end of the exit node can be deleted first. Figure 7 The output node shown is an AND node 607, and the external node it points to is the Reshape node 606. An arrow from AND node 607 points to Reshape node 606, indicating that the output of AND node 607 serves as the input of Reshape node 606. Now, the arrow between these two nodes, i.e., the pointing relationship, is removed.

[0145] Then, the process of deleting nodes with an out-degree of 0 in the static graph is repeated until there is no node with an out-degree of 0. The out-degree is the number of nodes pointed to by the output end of a node. Figure 7In the example, since the pointing relationship between the sum node 607 and the resynthesized shape node 606 is deleted, the output end of the sum node 607 no longer points to any node, so the out-degree becomes 0, and the sum node 607 is deleted. After the sum node 607 is deleted, the shard node 605 and the constant node 606 no longer point to any node, and the out-degree also becomes 0, so these two nodes are also deleted. After the shard node 605 is deleted, the extracted input tensor shape node 603 no longer points to any node, and the out-degree becomes 0, so the node is deleted. After the extracted input tensor shape node 603 is deleted, the output end of the input tensor node 601 still points to the convolution node 602, and the out-degree is not 0, so it cannot be deleted. At this time, the constant subgraph is considered to have been deleted. At this time, a constant node is constructed with the obtained constant, and the constructed constant node points to the external node pointed to by the output end of the exit node of the deleted constant subgraph, thereby replacing the constant subgraph. As shown in the following example Figure 7 As shown, constant node 609 is constructed using the obtained constants, pointing to the re-synthesized shape node 608 pointed to by the exit node of the constant subgraph (and node 607), and the constant subgraph is now equivalent to being replaced by constant node 609. In this way, constant nodes are constructed using the obtained constants, and the constant subgraph in the deep learning model is replaced by the constant nodes. The simplified model after replacing the constant subgraph in the deep learning model with the constant nodes is called the second deep learning model, that is, the second deep learning model after constant folding.

[0146] The value output by the exit node of the constant subgraph is actually the value output by the entire constant subgraph when all nodes in the constant subgraph work together under the premise that the input tensor of the deep learning model has a fixed shape. This value is still a constant. Therefore, a constant node is constructed using the determined constant. This constant node comprehensively reflects the overall effect of all nodes in the constant subgraph. It replaces the constant subgraph in the deep learning model without affecting the operation of the deep learning model. It can also further reduce the amount of data transmitted between the processing unit and the acceleration unit when the deep learning model is running, thereby improving the execution efficiency of the deep learning model.

[0147] Method flow of constant folding

[0148] like Figure 8 As shown, constant folding according to one embodiment of the present disclosure includes the following steps performed by the processing unit 220:

[0149] Step 710: extract a constant subgraph from the static graph of the deep learning model, wherein the constant subgraph is a connected subgraph consisting of nodes whose output values ​​are constants under the assumption that the input tensor of the deep learning model has a fixed shape;

[0150] Step 720: Input any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtain a constant output by an exit node of the constant subgraph, where the exit node refers to a node in the constant subgraph whose output terminal points to an external node of the constant subgraph and whose input terminal is an internal node of the constant subgraph;

[0151] Step 730: construct a constant node using the acquired constants, and use the constant node to replace the constant subgraph in the deep learning model to form a second deep learning model.

[0152] Optionally, step 710 includes:

[0153] Marking constant nodes and shape-related nodes that are only related to the shape of the input tensor of the deep learning model in the static graph;

[0154] Based on the marked constant nodes and shape-related nodes, the constant subgraph is extracted.

[0155] Optionally, extracting the constant subgraph based on the marked constant nodes and shape-associated nodes includes: repeatedly marking the nodes in the static graph whose input ends are all marked nodes until all nodes whose input ends are all marked nodes are marked; and dividing the interconnected nodes among the marked nodes into a constant subgraph.

[0156] Optionally, step 720 includes:

[0157] Constructing a tensor satisfying the fixed shape;

[0158] Inputting the constructed tensor as an input tensor into the static graph;

[0159] Get the constant output by the exit node of the constant subgraph.

[0160] Optionally, step 730 includes:

[0161] Deleting the constant subgraph;

[0162] A constant node is constructed using the obtained constant, where the constructed constant node points to the external node pointed to by the output terminal of the exit node of the deleted constant subgraph.

[0163] Optionally, deleting the constant subgraph includes:

[0164] Deleting the pointing relationship between the exit node and the external node pointed to by the output end of the exit node;

[0165] Repeat the process of deleting nodes with an out-degree of 0 in the static graph until no nodes with an out-degree of 0 exist, wherein the out-degree is the number of nodes pointed to by the output end of the node.

[0166] Since the implementation details of the above process have been described in detail in the description of the above device embodiment, they are not repeated here.

[0167] The commercial value of the disclosed embodiments

[0168] It has been verified through experiments that the amount of data transmitted between the processing unit and the acceleration unit when the deep learning model is running is only 60% of the solution in the prior art that only folds constant nodes. This greatly reduces the amount of data transmitted during model execution, improves the execution efficiency of the deep learning model, and has good market prospects.

[0169] It should be understood that the various embodiments in this specification are described in a progressive manner. References to the same or similar parts of the various embodiments will be sufficient. Each embodiment focuses on the differences from the other embodiments. In particular, the method embodiments are generally similar to the methods described in the device and system embodiments, so their description is relatively simple. For relevant details, references to the descriptions of the other embodiments will suffice.

[0170] It should be understood that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] It should be understood that an element described herein in the singular or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown herein as separate may be combined into a single module or element, and modules or elements described or shown herein as single may be split into multiple modules or elements.

[0172] It should also be understood that the terms and expressions used herein are for descriptive purposes only, and the one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or portions thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A processing unit for determining a model suitable for operation of an acceleration unit in a computing device, wherein: The processing unit includes: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit, configured to execute the decoded computer instructions to achieve: Extracting a constant subgraph from a static graph of a deep learning model, wherein the constant subgraph is a connected subgraph consisting of nodes whose output values ​​are constants assuming that an input tensor of the deep learning model has a fixed shape; Input any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtain a constant output by an exit node of the constant subgraph, where the exit node refers to a node in the constant subgraph whose output terminal points to an external node of the constant subgraph and whose input terminal is an internal node of the constant subgraph; Constructing a constant node using the obtained constants, and replacing the constant subgraph in the deep learning model with the constant node to form a second deep learning model, so as to reduce the amount of data transmitted between the processing unit and the acceleration unit when the model is actually running; The instruction execution unit is further configured to execute the decoded computer instructions to implement: after forming the second deep learning model, converting the second deep learning model into a universal intermediate representation; After converting the second deep learning model into a universal intermediate expression, inserting a maximum and minimum value collection node into the universal intermediate expression to collect the maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the maximum and minimum value collection node has been inserted back into a specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the quantization node and dequantization node have been inserted back into a specific format to obtain a quantized model; After obtaining the quantized model, converting the quantized model into an acceleration unit model according to an instruction set supported by the acceleration unit, and sending the model to the acceleration unit; Among them, the acceleration unit is a unit designed to accelerate the computing processing speed of the deep learning model.

2. The processing unit according to claim 1, wherein: The step of extracting a constant subgraph from a static graph of a deep learning model includes: Marking constant nodes and shape-related nodes that are only related to the shape of the input tensor of the deep learning model in the static graph; Based on the marked constant nodes and shape-related nodes, the constant subgraph is extracted.

3. The processing unit according to claim 2, wherein: The step of extracting the constant subgraph based on the marked constant nodes and shape-related nodes includes: Repeat the process of marking the nodes in the static graph whose input ends are all marked nodes until all the nodes whose input ends are all marked nodes are marked; Divide the interconnected nodes among the marked nodes into a constant subgraph.

4. The processing unit according to claim 1, wherein: Inputting any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtaining a constant output by an exit node of the constant subgraph, comprises: Constructing a tensor satisfying the fixed shape; Inputting the constructed tensor as an input tensor into the static graph; Get the constant output by the exit node of the constant subgraph.

5. The processing unit according to claim 1, wherein The obtained constants are used to construct constant nodes, and the constant subgraph in the deep learning model is replaced with the constant nodes to form a second deep learning model, including: Deleting the constant subgraph; A constant node is constructed using the obtained constant, where the constructed constant node points to the external node pointed to by the output terminal of the exit node of the deleted constant subgraph. The processing unit according to claim 5 , wherein: The deleting the constant subgraph includes: Deleting the pointing relationship between the exit node and the external node pointed to by the output end of the exit node; Repeat the process of deleting nodes with an out-degree of 0 in the static graph until no nodes with an out-degree of 0 exist, wherein the out-degree is the number of nodes pointed to by the output end of a node.

7. A computing device comprising: The processing unit according to any one of claims 1 to 6; a memory for storing the computer instructions; The acceleration unit is used to run the deep learning model.

8. A system on chip, comprising an acceleration unit for running a deep learning model, wherein the deep learning model is processed by the processing unit according to any one of claims 1 to 6 and then deployed on the acceleration unit.

9. A data center comprising the computing device according to claim 7.

10. A processing method for determining a model suitable for operation of an acceleration unit in a computing device, wherein: The method comprises: Extracting a constant subgraph from a static graph of a deep learning model, wherein the constant subgraph is a connected subgraph consisting of nodes whose output values ​​are constants assuming that an input tensor of the deep learning model has a fixed shape; Input any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtain a constant output by an exit node of the constant subgraph, where the exit node refers to a node in the constant subgraph whose output terminal points to an external node of the constant subgraph and whose input terminal is an internal node of the constant subgraph; Constructing a constant node using the obtained constants, and replacing the constant subgraph in the deep learning model with the constant node to form a second deep learning model, so as to reduce the amount of data transmitted between the processing unit and the acceleration unit when the model is actually running; After forming the second deep learning model, converting the second deep learning model into a universal intermediate representation; After converting the second deep learning model into a universal intermediate expression, inserting a maximum and minimum value collection node into the universal intermediate expression to collect the maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the maximum and minimum value collection node has been inserted back into a specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the quantization node and dequantization node have been inserted back into a specific format to obtain a quantized model; After obtaining the quantized model, converting the quantized model into an acceleration unit model according to an instruction set supported by the acceleration unit, and sending the model to the acceleration unit; Among them, the acceleration unit is a unit designed to accelerate the computing processing speed of the deep learning model.

11. The processing method according to claim 10, wherein: The step of extracting a constant subgraph from a static graph of a deep learning model includes: Marking constant nodes and shape-related nodes that are only related to the shape of the input tensor of the deep learning model in the static graph; Based on the marked constant nodes and shape-related nodes, the constant subgraph is extracted.

12. The processing method according to claim 11, wherein: The step of extracting the constant subgraph based on the marked constant nodes and shape-related nodes includes: Repeat the process of marking the nodes in the static graph whose input ends are all marked nodes until all the nodes whose input ends are all marked nodes are marked; Divide the interconnected nodes among the marked nodes into a constant subgraph.

13. The processing method according to claim 10, wherein: Inputting any tensor that satisfies the fixed shape as an input tensor into the static graph, and obtaining a constant output by an exit node of the constant subgraph, comprises: Constructing a tensor satisfying the fixed shape; Inputting the constructed tensor as an input tensor into the static graph; Get the constant output by the exit node of the constant subgraph.

14. The processing method according to claim 10, wherein: The obtained constants are used to construct constant nodes, and the constant subgraphs in the deep learning model are replaced with the constant nodes to form a second deep learning model, including: Deleting the constant subgraph; A constant node is constructed using the obtained constant, where the constructed constant node points to the external node pointed to by the output terminal of the exit node of the deleted constant subgraph.

15. The processing method according to claim 14, wherein: The deleting the constant subgraph includes: Deleting the pointing relationship between the exit node and the external node pointed to by the output end of the exit node; Repeat the process of deleting nodes with an out-degree of 0 in the static graph until no nodes with an out-degree of 0 exist, wherein the out-degree is the number of nodes pointed to by the output end of a node.

Images