Processing unit, computing device, and computational graph optimization method for deep learning model

Abstract

The present invention discloses a processing unit, a computing device, and a method for optimizing a computational graph of a deep learning model. The method includes: determining multiple paths extending from an input operator to an output operator of a computational graph to be optimized and containing at least one transposition operator, wherein each of the multiple paths has at least one transposition operator different from the other paths; determining a merging direction for each transposition operator in each path; and determining, based on the merging direction of each transposition operator in each path, transposition operators that can be offset in one or more paths, and removing the transposition operators that can be offset in one or more paths to obtain an optimized computational graph. The optimized computational graph removes unnecessary transposition operators within the computational graph, thereby reducing resource overhead.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more specifically, to a processing unit, a computing device, and a computational graph optimization method for a deep learning model. Background Art

[0002] In recent years, with the continued rise of artificial intelligence, deep learning models have become a widely used predictive model. Deep learning models are increasingly being applied in various scenarios, such as speech recognition and facial recognition. Cloud technology is a managed technology that unifies hardware, software, network, and other resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Using cloud technology, customers can provide the computational graph of their desired deep learning model to a cloud service provider, who then processes the model and deploys it to servers in a data center. Customers' application systems can then use the data center's deep learning model computational graph to generate predictions. Because data centers use dedicated acceleration units to run deep learning model computational graphs, deploying deep learning models in data centers can help improve their inference performance.

[0003] In the process of optimizing the computational graph of the deep learning model that the customer wants to use, the inventor discovered that some computational graphs contain unnecessary transposition (trans) operators. The transposition operator is used to convert the data format of the tensor (also known as dimensionality conversion). The transposition operator is executed by the acceleration unit or the processor. When the acceleration unit is capable of executing the transposition operator, the transposition operator is executed by the acceleration unit, and when the acceleration unit cannot execute the transposition operator, the transposition operator is executed by the processor. The former occupies the resources and time of the acceleration unit, and the latter requires switching between the processor and the acceleration unit, thus bringing greater delay. Therefore, unnecessary transposition operators bring unnecessary resource overhead. In order to reduce unnecessary resource overhead, it should be ensured that there are no unnecessary transposition operators in the computational graph of the deep learning model to be run on the acceleration unit. Summary of the Invention

[0004] Based on this, the purpose of the present disclosure is to provide a computational graph optimization method for a processing unit, a computing device, and a deep learning model to eliminate unnecessary transposition operators in the computational graph.

[0005] In a first aspect, an embodiment of the present disclosure provides a processing unit, including:

[0006] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0007] an instruction decoding unit for decoding the retrieved computer instructions;

[0008] An instruction execution unit, configured to execute the decoded computer instructions to achieve:

[0009] Determine a plurality of paths extending from an input operator to an output operator of the computation graph to be optimized and containing at least one transposition operator, wherein each of the plurality of paths has at least one transposition operator different from other paths;

[0010] Determine the merge direction of each transpose operator in each path; and

[0011] The transposition operators that can be offset in one or more paths are determined according to the merging direction of each transposition operator in each path, and the transposition operators that can be offset in the one or more paths are cleared to obtain an optimized computation graph.

[0012] Optionally, the instruction execution unit further implements:

[0013] Get the initial computation graph;

[0014] Determine whether the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit. If the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit, use the initial calculation graph as the calculation graph to be optimized; if the data format of the initial calculation graph is different from the data format selected by the specific acceleration unit, insert a first transposition operator and a second transposition operator in sequence before the input operator of the initial calculation graph, and insert a third transposition operator and a fourth transposition operator in sequence after the output operator of the initial calculation graph, and use the initial calculation graph after the transposition operators are inserted as the calculation graph to be optimized, wherein the first transposition operator and the third transposition operator are used to convert the data format of the calculation graph to be optimized into the data format selected by the specific acceleration unit, and the second transposition operator and the fourth transposition operator are used to convert the data format selected by the specific acceleration unit into the data format of the calculation graph to be optimized.

[0015] Optionally, determining the merging direction of each transposition operator in each path includes:

[0016] For each transposition operator in each path in the path set, first determine whether the data format of the output tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is upward. If not, then continue to determine whether the data format of the input tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is downward.

[0017] Then, determining the cancelable transposition operators in one or more paths according to the merging direction of each transposition operator in each path, and clearing the cancelable transposition operators in the one or more paths includes:

[0018] Traversing each path in turn, for each adjacent first transposition operator and second transposition operator whose merging directions are downward and upward respectively on the path, determining whether the first transposition operator and the second transposition operator only exist in exactly the same path; if so, determining that the first transposition operator and the second transposition operator are cancelable transposition operators, and clearing the first transposition operator and the second transposition operator;

[0019] If not, if it is determined that the first transposition operator exists alone in at least one path, and if it is determined that the merging direction of the subsequent transposition operator adjacent to the first transposition operator in all paths containing the first transposition operator is upward, then the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are determined to be cancelable transposition operators, and the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are cleared in all paths containing the first transposition operator; if it is determined that the second transposition operator exists alone in one path, and it is determined that the merging direction of the previous transposition operator adjacent to the second transposition operator in all paths containing the second transposition operator is downward, then the second transposition operator and the previous transposition operator adjacent to the second transposition operator are determined to be cancelable transposition operators, and the second transposition operator and the previous transposition operator adjacent to the second transposition operator are cleared in all paths containing the second transposition operator.

[0020] Optionally, the data format of the computation graph to be optimized and the data format selected by the specific acceleration unit are one of the following data formats: NHWC and NCHW.

[0021] Optionally, the instruction execution unit further implements: slicing the complete computational graph of a specific deep learning model to obtain multiple subgraphs, and using one of the subgraphs as the computational graph to be optimized.

[0022] Optionally, the instruction execution unit further implements: converting the complete computational graph of a specific deep learning model into an intermediate expression that complies with the requirements of the specific acceleration unit, performing at least one of operator merging, quantization, and model pruning on the intermediate expression, cutting the processed intermediate expression into multiple subgraphs, and using one of the subgraphs as the computational graph to be optimized.

[0023] Optionally, the computation graph to be optimized is a complete computation graph of a specific deep learning model.

[0024] Optionally, the step of determining multiple paths includes:

[0025] Searching for all paths extending from an input operator to an output operator of the computation graph to be optimized and containing at least one transposition operator, and forming all paths into a path set;

[0026] For each path, it is determined one by one whether the transposition operator on the path is the same as the transposition operators on other paths. If there are different transposition operators, the path is retained in the path set.

[0027] Optionally, the step of determining multiple paths includes:

[0028] Search each path extending from the input operator to the output operator of the computation graph to be optimized, then determine whether there is a transposition operator on the path and whether the path has a different transposition operator from the existing paths in the path set. If both are true, store the path in the path set.

[0029] Optionally, the merging direction of each transposition operator is determined according to the tensor dimension transformation performed by the transposition operator.

[0030] In a second aspect, an embodiment of the present disclosure provides a computing device comprising an acceleration unit, a memory, and any one of the processing units described above.

[0031] In a third aspect, an embodiment of the present disclosure provides a method for optimizing a computational graph of a deep learning model, including:

[0032] Determine a plurality of paths extending from an input operator to an output operator of the computation graph to be optimized and containing at least one transposition operator, wherein each of the plurality of paths has at least one transposition operator different from other paths;

[0033] Determine the merge direction of each transpose operator in each path; and

[0034] The transposition operators that can be offset in one or more paths are determined according to the merging direction of each transposition operator in each path, and the transposition operators that can be offset in the one or more paths are cleared to obtain an optimized computation graph.

[0035] Optionally, it also includes:

[0036] Get the initial computation graph;

[0037] Determine whether the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit. If the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit, use the initial calculation graph as the calculation graph to be optimized; if the data format of the initial calculation graph is different from the data format selected by the specific acceleration unit, insert a first transposition operator and a second transposition operator in sequence before the input operator of the initial calculation graph, and insert a third transposition operator and a fourth transposition operator in sequence after the output operator of the initial calculation graph, and use the initial calculation graph after the transposition operators are inserted as the calculation graph to be optimized, wherein the first transposition operator and the third transposition operator are used to convert the data format of the calculation graph to be optimized into the data format selected by the specific acceleration unit, and the second transposition operator and the fourth transposition operator are used to convert the data format selected by the specific acceleration unit into the data format of the calculation graph to be optimized.

[0038] Optionally, determining the merging direction of each transposition operator in each path includes:

[0039] For each transposition operator in each path in the path set, first determine whether the data format of the output tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is upward. If not, then continue to determine whether the data format of the input tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is downward.

[0040] Then, determining the cancelable transposition operators in one or more paths according to the merging direction of each transposition operator in each path, and clearing the cancelable transposition operators in the one or more paths includes:

[0041] Traversing each path in turn, for each adjacent first transposition operator and second transposition operator whose merging directions are downward and upward respectively on the path, determining whether the first transposition operator and the second transposition operator only exist in exactly the same path; if so, determining that the first transposition operator and the second transposition operator are cancelable transposition operators, and clearing the first transposition operator and the second transposition operator;

[0042] If not, if it is determined that the first transposition operator exists alone in at least one path, and if it is determined that the merging direction of the subsequent transposition operator adjacent to the first transposition operator in all paths containing the first transposition operator is upward, then the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are determined to be cancelable transposition operators, and the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are cleared in all paths containing the first transposition operator; if it is determined that the second transposition operator exists alone in one path, and it is determined that the merging direction of the previous transposition operator adjacent to the second transposition operator in all paths containing the second transposition operator is downward, then the second transposition operator and the previous transposition operator adjacent to the second transposition operator are determined to be cancelable transposition operators, and the second transposition operator and the previous transposition operator adjacent to the second transposition operator are cleared in all paths containing the second transposition operator.

[0043] Optionally, the data format of the computation graph to be optimized and the data format selected by the specific acceleration unit are one of the following data formats: NHWC and NCHW.

[0044] Optionally, the method further includes: before the step of determining whether the data format of the computation graph to be optimized is the same as the data format selected by the specific acceleration unit,

[0045] The complete computational graph of a specific deep learning model is sliced to obtain multiple subgraphs, and one of the subgraphs is used as the computational graph to be optimized.

[0046] Optionally, the method further includes: before the step of determining whether the data format of the computation graph to be optimized is the same as the data format selected by the specific acceleration unit,

[0047] The complete computational graph of a specific deep learning model is converted into an intermediate expression that complies with the specific acceleration unit, the intermediate expression is subjected to at least one of operator merging, quantization, and model pruning, the processed intermediate expression is sliced to obtain multiple subgraphs, and one of the subgraphs is used as the computational graph to be optimized.

[0048] Optionally, the computation graph to be optimized is a complete computation graph of a specific deep learning model.

[0049] Optionally, the step of determining multiple paths includes:

[0050] Searching for all paths extending from an input operator to an output operator of the computation graph to be optimized and containing at least one transposition operator, and forming all paths into a path set;

[0051] For each path, it is determined one by one whether the transposition operator on the path is the same as the transposition operators on other paths. If there are different transposition operators, the path is retained in the path set.

[0052] Optionally, the step of determining multiple paths includes:

[0053] Search each path extending from the input operator to the output operator of the computation graph to be optimized, then determine whether there is a transposition operator on the path and whether the path has a different transposition operator from the existing paths in the path set. If both are true, store the path in the path set.

[0054] In a fourth aspect, this public embodiment provides a data center, comprising any computing device described above.

[0055] The computation graph optimization method of the disclosed embodiment first determines the merging direction of each transposition operator in each path on the computation graph. Then, based on the merging direction of each transposition operator, it determines whether adjacent transposition operators in each path can be offset. Based on this, a transposition operator clearing operation is performed to obtain an optimized computation graph. The optimized computation graph reduces resource overhead by eliminating unnecessary transposition operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0057] Figure 1 A hierarchical structure diagram of a data center to which an embodiment of the present disclosure is applied is shown;

[0058] Figure 2 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0059] Figure 3 This is a block diagram of the internal structure of a server in a data center according to an embodiment of the present disclosure;

[0060] Figure 4 This is a control relationship diagram of a central processing unit (CPU) and an acceleration unit within a server according to an embodiment of the present disclosure;

[0061] Figure 5 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present disclosure;

[0062] Figure 6 It is a layered software architecture diagram;

[0063] Figure 7 is an example graph of computational graph transformation;

[0064] Figure 8a and 8b is a flowchart of a computation graph optimization method provided by one embodiment and another embodiment of the present disclosure;

[0065] Figure 9a and 9b yes Figure 8a The combination of steps S802 and S803 and Figure 8b A flowchart of an embodiment of a combination of steps S815 and S816;

[0066] Figures 10a-10c This is an example obtained according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0068] The following terms are used in this article.

[0069] Accelerator unit: A processing unit designed to improve data processing speed in specialized areas (e.g., image processing, neural network operations, etc.) where general-purpose processors are inefficient. It is often used in conjunction with a general-purpose processor (CPU), controlled by the general-purpose processor, to perform processing for specific purposes or fields, thereby improving computer processing efficiency in those areas. It can also be called an AI processing unit and can include a graphics processing unit (GPU), a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated AI acceleration hardware (e.g., an accelerator unit).

[0070] On-chip memory: Memory that is used exclusively within the main core or sub-core and cannot be shared.

[0071] The command processor serves as the command interface between the accelerator unit and the central processing unit (CPU), which drives it. The command processor receives instructions from the CPU for the accelerator unit and distributes them to the various cores within the accelerator unit for execution. It also manages the synchronization of the cores within the accelerator unit.

[0072] Lifecycle: An operand isn't always involved in the entire instruction sequence. The period between its first appearance and its last use in the instruction sequence is the lifecycle of the operand. In other words, after its lifecycle, it's no longer needed and doesn't need to be stored in on-chip memory.

[0073] Neural network: Generally refers to an artificial neural network (ANN), an algorithmic network that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. A classic neural network, also the simplest neural network structure, consists of three layers: input layer, output layer, and middle layer (also called hidden layer). The input layer, output layer, and middle layer each contain multiple nodes.

[0074] Neural network model: In a neural network, nodes are mathematized to generate mathematical models of nodes. The mathematical models of a large number of nodes in a neural network constitute the neural network model.

[0075] Deep learning model: The concept of deep learning originates from the study of neural networks. A neural network with multiple intermediate layers is called a deep learning network. Therefore, in this sense, a deep learning model is also a type of neural network model. Both deep learning models and neural network models must be trained. Sample data is input into a designed network structure (i.e., the network structure has been determined), feature information is extracted through multiple intermediate layers, and the weight parameters of the neurons are continuously modified based on the output results of the output layer, so that the output results of the output layer increasingly converge to the preset results until the final weight parameters are determined. The trained deep learning model can be truly applied in real-world scenarios. At the same time, the use of the deep learning model in real-world scenarios can also be collected to optimize the deep learning model.

[0076] Node: The smallest unit of independent operation in a deep learning model, which receives input and generates output after operating on its own weight parameters or parameters in other models (such as hyperparameters). A deep learning model can include various specific operations such as convolution and pooling, and has various operation nodes including convolution nodes and pooling nodes. There are multiple layers in a deep learning model, each layer has multiple nodes, and the output of each node is the input of the node in the next layer. Specifically, a node includes a program for a specific operation and related data. For example, a convolution operation node includes the program code used for the convolution operation and some data used in the convolution.

[0077] An operator is a collection of operations built into a deep learning model to implement specific functionality. Each layer of a deep learning model can contain multiple such operators. These are referred to as operations in the TensorFlow framework and layers in the Caffe framework. Operators are considered a further abstraction based on nodes; an operator can correspond to one or more nodes. Therefore, operators and nodes can sometimes represent the same program code.

[0078] Instruction set: A collection of instructions for operations supported within the chip, such as operations that mainly support deep learning operators, such as Convolution, Pooling, ROI, etc.

[0079] Quantization: The act of converting the inputs of operation nodes in a deep learning model, as well as the weight parameters and other parameters of the operation nodes, from high-precision data types to low-precision data types, thereby reducing the requirements for data throughput and storage space.

[0080] Dequantization: The opposite process of quantization, which is the process of converting the input of the operation node in the deep learning model, as well as the weight parameters and other parameters of the operation node from low-precision data types to high-precision data types.

[0081] Intermediate Representation (IR): Deep learning models have different formats, depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet. The code representations of these deep learning models also vary. This poses significant challenges to the universality of deep learning model quantization. An intermediate representation (IR) converts deep learning model code in various formats into a single representation that conforms to one or more acceleration units. It analyzes the meaning of each code statement in a deep learning model and translates it into a common representation based on its meaning. This ensures that code statements with the same meaning across different deep learning models are expressed identically in the IR. Currently, there are tools available that convert the representations of different deep learning models into IR.

[0082] Computation graph: Currently, deep learning frameworks mainly have two programming methods: declarative programming and imperative programming. In declarative programming, the program code first defines a neural network model structure to describe the calculation logic, but it is not executed immediately. The neural network model structure is only executed when the program code that calls the network model structure is executed. The neural network model structure includes multiple operators (or symbolic expressions of operators) and the connection relationship between them, and can be shown graphically. Therefore, the neural network model structure is called a static computation graph. In imperative programming, the program code directly returns the result of the operation, and the definition and execution of the neural network model structure are synchronized. Generally speaking, static graphs are convenient for compiling and optimizing the overall neural network model, which is more conducive to performance improvement, while dynamic graphs are very convenient for users to debug specific programs.

[0083] Figure 1 A hierarchical structure diagram of a data center is shown as a scenario in which an embodiment of the present disclosure is applied.

[0084] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Neural networks, a key AI technology, are already widely used in data center big data analytics.

[0085] In traditional large data centers, the network structure is usually Figure 1 The three-layer structure shown is a hierarchical inter-networking model. This model consists of the following three layers:

[0086] Access Layer 103: Sometimes also called the edge layer, it includes access switches 130 and the servers 140 connected to them. Servers 140 are the processing and storage entities of the data center, processing and storing large amounts of data within the data center. Access switches 130 connect these servers to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, hence the name "top of rack" switches, physically connecting the servers.

[0087] Aggregation Layer 102: sometimes also called the distribution layer, includes aggregation switches 120. Each aggregation switch 120 connects to multiple access switches and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0088] Core Layer 101: Includes core switch 110. Core switch 110 provides high-speed forwarding for packets entering and exiting the data center and provides connectivity for multiple aggregation layers. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 provides a resilient L3 routing network for the entire data center network.

[0089] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0090] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with the remaining aggregation switches 120 being used only in the event of a failure (as indicated by the dotted lines in the diagram above). This means that horizontal scalability is impossible at the aggregation layer because, even if multiple aggregation switches 120 are added, only one is still operational.

[0091] Figure 2 Shown Figure 1 The physical connection of each component in the hierarchical data center. Figure 2 As shown, one core switch 110 is connected to multiple aggregation switches 120 , one aggregation switch 120 is connected to multiple access switches 130 , and one access switch 130 is connected to multiple servers 140 .

[0092] server

[0093] Since server 140 is the real device in the data center, Figure 3The following diagram shows the internal structure of a server 140. Server 140 includes a bus-connected memory 210, a central processing unit (CPU) 220, and various acceleration units. These acceleration units include an acceleration unit 230, a data transfer unit (DTU) 260, a graphics processing unit (GPU, not shown), an application-specific integrated circuit (ASIC, not shown), and a field-programmable gate array (FPGA, not shown).

[0094] The traditional processor architecture design results in control units and storage units occupying a large portion of the space, while the computing unit occupies less space. As a result, while highly effective for logic control, it is inefficient for large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively increase the speed of computation for different functions and fields. The acceleration unit proposed in this disclosure can be any of these types, and each of these acceleration units is described below.

[0095] Accelerator unit 230: This is a processing unit that utilizes a data-driven parallel computing architecture to handle the large number of operations (e.g., convolution, pooling, etc.) within each neural network node. Because the data and intermediate results from these operations (e.g., convolution, pooling, etc.) are closely linked and frequently used throughout the computational process, existing CPU architectures require frequent access to external memory due to the limited internal memory capacity of the CPU core, resulting in inefficient processing. By implementing an acceleration unit, each core has on-chip memory suitable for the storage capacity required for neural network computations, eliminating the need for frequent access to external memory, significantly improving processing efficiency and computational performance.

[0096] Data Transfer Unit (DTU) 260: This is a wireless terminal device specifically designed to convert serial port data into IP data or vice versa for transmission over a wireless communication network. The DTU's primary function is to wirelessly transmit data from remote devices back to the backend center. At the front end, the DTU and the client's device are connected via an interface. After powering on, the DTU first registers with the mobile GPRS network and then establishes a socket connection with the backend center installed in the DTU. The backend center acts as the server of the socket connection, and the DTU is the client of the socket connection. Therefore, the DTU is used in conjunction with the backend software. Once the connection is established, the front-end device and the backend center can perform wireless data transmission via the DTU.

[0097] A graphics processing unit (GPU) is a microprocessor specifically designed for image and graphics-related computing. The GPU addresses the CPU's lack of computing space by utilizing a large number of dedicated graphics processing units. This reduces the graphics card's reliance on the CPU and allows it to handle some of the computationally intensive image processing tasks previously handled by the CPU.

[0098] Application-Specific Integrated Circuit (ASIC): An integrated circuit designed and manufactured to meet specific user requirements and the needs of a specific electronic system. Because this type of integrated circuit is customized according to user requirements, its structure is often adapted to that specific user's needs.

[0099] Field Programmable Gate Array (FPGA): A further development of programmable devices such as PALs and GALs. It emerged as a semi-custom circuit within the field of Application-Specific Integrated Circuits (ASICs), addressing both the shortcomings of custom circuits and the limited number of gates inherent in existing programmable devices.

[0100] Although the acceleration unit has the advantage of being much more efficient than an ordinary processor for specific applications or fields, it is also subject to the control of the processing unit 220. Taking the acceleration unit dedicated to deep learning models as an example, the memory 210 stores various deep learning models, including the neurons and neuron weight data of these models. These deep learning models are called when needed. Figure 3 A processing unit 220 is deployed to an acceleration unit 230. Specifically, the processing unit 220 can inform the acceleration unit 230 of the storage location of the deep learning model in memory 210 in the form of instructions. The acceleration unit 230 can then address these locations and store the instructions to be executed in its on-chip memory. The processing unit 220 can also send instructions to the acceleration unit 230 in the form of instructions. The acceleration unit 230 receives the instructions and stores them in the on-chip memory. Similarly, the acceleration unit 230 can also obtain input data in the above manner. The acceleration unit 230 obtains the instructions to be executed and the input data to perform inference calculations. The node weight parameters can be included in the instruction sequence of the deep learning model and retrieved from memory 210 by the acceleration unit 230. Of course, the node weight parameters can also be stored independently and retrieved from memory 210 by the acceleration unit 230 when needed. Here, the processing unit 220 can be understood as a hardware unit with scheduling and control capabilities, generally a hardware unit such as a central processing unit (CPU), microcontroller, or microprocessor.

[0101] Internal structure of the processing unit and acceleration unit 230

[0102] The following combination Figure 4The internal structure diagram of the processing unit and the acceleration unit 230 is shown, which illustrates how the processing unit controls the operation of the acceleration unit.

[0103] like Figure 4 As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0104] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 4 A register in the register file 229 shown is used to store instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, incrementing the address or decrementing the address according to the instruction length.

[0105] After fetching the instruction, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0106] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0107] After the instruction issue unit 225 sends the instruction to the instruction execution unit 226, the instruction execution unit 226 begins executing the instruction. However, if the instruction execution unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is for neural network inference, the instruction execution unit 226 will not execute the instruction but instead send it via the bus to the acceleration unit 230 for execution.

[0108] The acceleration unit 230 includes multiple cores 236 ( Figure 4 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0109] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus. According to different mechanisms, the bus channel 231 may include a PCIE channel 232, an I2C channel 233, and a JTAG channel 234.

[0110] PCIE, or PCI-Express, is a high-speed serial computer expansion bus standard proposed by Intel in 2001 to replace the old PCI, PCI-X, and AGP bus standards. PCIE is a high-speed serial point-to-point dual-channel high-bandwidth transmission. The connected devices are allocated exclusive channel bandwidth and do not share bus bandwidth. It mainly supports active power management, error reporting, end-to-end reliable transmission, hot plugging, and quality of service functions. Its main advantage is its high data transmission rate, and it also has considerable development potential. At present, most PCIE buses are PCIE GEN3, but the disclosed embodiments can also use PCIE GEN4, that is, a bus channel that complies with the PCI-Express 4.0 standard.

[0111] I2C channel 233 is a simple, bidirectional, two-wire synchronous serial bus developed by Philips. It only requires two wires to transmit information between devices connected to the bus.

[0112] JTAG stands for Joint Test Action Group and is the common name for IEEE Standard 1149.1, Standard Test Access Port and Boundary-Scan Architecture. This standard is used to verify the functionality of designed and manufactured printed circuit boards. JTAG was formally standardized in 1990 by IEEE Document 1149.1-1990. In 1994, a supplementary document describing the Boundary-Scan Description Language (BSDL) was added. Since then, this standard has been widely adopted by electronics companies worldwide. Boundary scan has become almost synonymous with JTAG. JTAG Channel 234 is a bus channel that complies with this standard.

[0113] The direct memory access (DMA) mechanism 235 is a function provided by some computer bus architectures that enables data to be written directly from an attached device (such as an external memory) to the on-chip memory of the acceleration unit 230. Compared to the method in which all data transmission between devices must pass through the command processor 237, this method greatly improves the efficiency of data access. It is precisely because of this mechanism that the core of the acceleration unit 230 can directly access the memory 210 and read parameters in the deep learning model (such as the weight parameters of each node), etc., greatly improving data access efficiency. Although the direct memory access mechanism 235 is shown in the figure as being located between the processor 237 and the bus channel 231, the design of the acceleration unit 230 is not limited to this. In some hardware designs, each acceleration unit core 236 can include a direct memory access mechanism 235, so that the acceleration unit core 236 does not need to go through the command processor 237, but directly reads data from the attached device and writes it to the on-chip memory of the acceleration unit 230.

[0114] The command processor 237 assigns instructions sent from the processing unit 220 to the acceleration unit 230 to the core 236 for execution. The instruction execution unit 226 sends pending instructions that require execution by the acceleration unit 230 to the acceleration unit 230, or the instruction execution unit 226 informs the acceleration unit 230 of the storage location of the pending instructions in the memory 210. After the pending instruction sequence enters the bus channel 231, it is cached in the command processor 237, which selects a core 236 and assigns the instruction sequence to it for execution. The pending instructions come from the compiled deep learning model. It should be understood that the pending instruction sequence can include pending instructions executed by the processing unit 220 and pending instructions that need to be executed by the acceleration unit 230.

[0115] Acceleration unit core

[0116] Figure 5 FIG. 4 is a diagram showing the internal structure of a core of an acceleration unit according to an embodiment of the present disclosure.

[0117] In one embodiment, Figure 5 As shown, core 236 includes a tensor engine 310, a pooling engine 320, a convolution process 330, an activation operation 380, a sequencer 350, an instruction cache 340, an on-chip memory 360, and a constant buffer 370. Among them, tensor engine 310, pooling engine 320, convolution process 330, and activation operation 380 are all hardware execution units. Hardware execution units are the hardware modules that actually perform various operations. Some other hardware execution units are not shown in the figure.

[0118] The instruction sequence assigned by the command processor 237 to the core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and distributes them to the various hardware execution units for execution based on the nature of the instructions. The tensor engine 310 is responsible for processing tensor-related operations in deep learning models. The pooling engine 320 is responsible for processing pooling operations in deep learning models. The convolution processing 330 is responsible for convolution operations in deep learning models. The activation operation 380 is used to perform operations corresponding to activation functions in deep learning models. The sequencer 350 determines the distribution of the instructions to the various hardware execution units based on the nature of the fetched instructions, such as convolution, matrix multiplication, or pooling.

[0119] The on-chip memory 360 is an in-core memory that stores the weight parameters in the deep learning model, as well as the inputs and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning model (for example, hyperparameters in the deep learning model) in addition to the weight parameters. As described above, when the processing unit 220 pre-configures the deep learning model in the acceleration unit 230, the processing unit 220 can send the location of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the acceleration unit 230 will retrieve them from the corresponding location in the memory 210 when needed and place them in the on-chip memory 360. For other parameters, the acceleration unit 230 will retrieve them from the corresponding location in the memory 210 when needed and place them in the constant buffer 370. In addition, when the executable instruction is assigned to the core 236 for execution by the command processor 237, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0120] Software architecture diagram

[0121] Improvements to deep learning models require not only the aforementioned hardware support but also continuous improvements at the software and algorithm levels. Only by combining underlying hardware support with the deep learning algorithm architecture above can a powerful computing engine be delivered.

[0122] Figure 6 This is a diagram of a layered software architecture. Layered software design is the mainstream design approach for large-scale software projects. It reduces dependencies between layers, allowing developers to focus on a single layer within the overall structure and easily replace existing layer implementations with new code.

[0123] As shown in the figure, from top to bottom, the software architecture diagram includes an application layer 401, a framework layer 402 and a function layer.

[0124] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, and recommendation 407. These applications are built using this architecture and can also call the architecture to provide runtime interfaces to gain reasoning capabilities within the application.

[0125] The framework layer 402 integrates various deep learning frameworks, such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to enable continued optimization and improvement of various algorithms. TensorFlow 408 is a symbolic mathematics system based on data flow programming and is widely used in programming various machine learning algorithms. MXNet 409 is the deep learning library of choice for Amazon. Caffe 410, or Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modularity.

[0126] The functional layer includes a compilation stack 403 and a runtime stack 404. The compilation stack 403 is used for various model conversions (converter 411), quantization 412, optimization 413, and compilation 414. Conversion 411 converts the model's internal data into an intermediate representation (IR) format. Quantization 412 converts parameters such as weights in a deep learning model, as well as inputs to the deep learning model, from high-precision data types to low-precision data types. Optimization 413 fuses operators within the model and performs operations such as multi-model optimization linking. Compilation 414 optimizes the model based on the hardware, generating a binary model that the hardware can recognize. The runtime stack 404 includes a runtime API 415, an execution manager 416, a user-mode driver 417, and a kernel-mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized runtime API 415 provides various runtime callable interfaces. The user-mode driver 417 provides kernel-mode hardware commands and resource scheduling. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0127] By integrating multiple mainstream deep learning models into a single open source platform, developers can develop, compile, and run multiple deep learning models on the same platform, eliminating the need to deploy and maintain multiple model frameworks. Furthermore, the open source platform can be expanded to support more deep learning models.

[0128] The following describes the computational graph processing process in the actual R&D process based on the above software architecture diagram.

[0129] refer to Figure 7 As shown. First, as shown in the figure, various deep learning frameworks support models A to M, and the first computational graph 701 of a specific deep learning framework is converted into an intermediate expression 702. The intermediate expression 702 predetermines many operators and their attributes. Converting the first computational graph 701 into the intermediate expression 702 includes: each operator and its attributes in the first computational graph 701 are converted into the corresponding operator and its attributes defined by the intermediate expression 702. This conversion can be achieved with the help of conversion 411. Conversion 411 defines a mapping function to achieve operator conversion. The mapping function is used to convert between operators with the same function but different attributes. Developers understand the specific functions of each operator and their respective attribute definitions, and predefine mapping functions for operators with the same function but different attributes. Referring to the operator mapping table shown in Table 1, the left column is the operator identifier of the first operator, and the right column is the name of the mapping function.

[0130] Table 1

[0131]

[0132] Operator attribute conversion is not limited to the use of mapping functions, and other methods can also be used to complete operator attribute conversion.

[0133] The intermediate representation 702 is then subjected to model processing such as operator merging, model pruning, quantization, and graph slicing, outputting a second computational graph 703. Quantization inserts quantization and dequantization nodes into the intermediate representation 702. Operator merging combines two or more operators into a single operator based on the hardware. Graph slicing divides the intermediate representation into several subgraphs to facilitate reading and processing by the acceleration unit. Model pruning is a model compression method that introduces sparsity to the dense connections of a deep learning model, reducing the number of non-zero weights by directly setting "unimportant" weights to zero. The second computational graph 703 can be deployed for execution on a designated acceleration unit.

[0134] Finally, if necessary, the second computation graph 703 must be converted back to a computation graph supported by a specific deep learning framework. This specifically involves converting all operators and their attributes in the second computation graph 703 into operators and their attributes supported by the specific deep learning framework. This conversion can also be achieved using a mapping function. Of course, this step can also be implemented in other ways. In addition, since the computation graph contains various operators and the connections between them, after the operator conversion, it is necessary to ensure that the connections between the operators are correct, or the connections between the operators can be directly reconstructed.

[0135] Data Format and Transpose Operator

[0136] A tensor is a generalization of scalars, vectors, and matrices. Broadly speaking, tensors include scalars, vectors, two-dimensional matrices, and matrices of three or more dimensions (i.e., high-dimensional matrices). The data format describes how the raw data is arranged within the tensor. Let's take image data as an example. Generally speaking, the raw image data input to a deep learning model is a batch of images, represented by a four-dimensional matrix. The data format of the four-dimensional matrix is represented by a one-dimensional array, which specifies how the image data is organized and stored. Knowing the data format is crucial for correctly reading the image data. Each deep learning framework typically supports one or more data formats. For example, some deep learning frameworks use the four characters 'NHWC' to indicate the supported data formats, where N represents the number of images contained in the image data, H represents the number of vertical pixels per image, W represents the number of horizontal pixels per image, and C represents the number of channels (e.g., 3 for an RGB image). When the data format is specified as 'NCHW', the number of channels C is placed in the outer layer of the tensor, and the pixels within each channel are closely packed together, resulting in a tensor format similar to 'RRRRRRGGGGGGBBBBBB'. When the data format is specified as 'NHWC', C is placed in the innermost layer, and the pixels corresponding to the spatial positions of multiple channels are closely packed together, resulting in a tensor format similar to 'RGBRGBRGBRGBRGBRGB'.

[0137] The transpose operator is used to convert the data format of a tensor (convert the tensor's dimensions). The transpose operator takes two input parameters: the input tensor and a control parameter, which converts the tensor's dimensions based on the control parameter. The following hypothetical example illustrates the operation of the transpose operator. Assume that the input tensor is A = [[[0,1,2,3],[4,5,6,7],[8,9,10,11]],[[12,13,14,15],[16,17,18,19],[20,21,22,23]]], and the control parameter perm = [0,2,1], indicating that the tensor A is to be converted from a 2*3*4 dimension to a 2*4*3 dimension. The input tensor A can be viewed as two 3*4 matrices A1 and A2, as shown below.

[0138]

[0139] The transpose operator converts tensor A from 2*3*4 dimensions to 2*4*3 dimensions, which means converting tensor A1 from 3*4 dimensions to 4*3 dimensions, and converting A2 from 3*4 dimensions to 4*3 dimensions, as shown below.

[0140]

[0141]

[0142] Finally, C1 and C2 are combined to obtain a 2*4*3 tensor C=[[[0,4,8],[1,5,9],[2,6,10],[3,7,11]],[[12,16,20],[13,17,21],[14,18,22],[15,19,23]]].

[0143] Considering the image data described above, let's assume that the data format of a tensor T1 is 'NCHW', where T1 = [[R,R,R,R,R,R],[G,G,G,G,G,G],[B,B,B,B,B,B]], and the data format of a tensor T2 is 'NHWC', where T2 = [[R,G,B],[R,G,B],[R,G,B],[R,G,B],[R,G,B],[R,G,B]], and we can use the transpose operator to convert the rows and columns of T1 to obtain tensor T2. Thus, the transpose operator converts the data format of the tensor.

[0144] The technical solution of the embodiment of the present disclosure

[0145] As described in the background, customers provide cloud service providers with the computational graphs for deep learning models to be run in their data centers. However, prior to this, customers typically perform various processing on the computational graphs to adapt them to the currently running acceleration units. For example, the data format of the computational graphs may be adjusted. Those skilled in the art understand that different acceleration units may use different data formats. For example, a graphics processing unit (GPU) may use 'NCHW' to process large amounts of image data and therefore prefers contiguous pixels within the same channel, while another acceleration unit may use 'NHWC'. Therefore, when deploying a computational graph to a specific acceleration unit, developers determine whether the graph's data format (i.e., the data format of the graph's input tensors) is the same as the data format used by the specific acceleration unit. If they are different, transpose operators are inserted before and after the computational graph. The transpose operator at the beginning of the graph converts the input tensors to the data format selected by the specific acceleration unit, while the transpose operator at the end of the graph converts the output tensors back to the original data format. Furthermore, when developers need to test a subgraph on a terminal's processor, transposition operators are inserted to ensure normal execution of the subgraph. These operations can cause the computational graph of the deep learning model provided by the customer to the cloud service provider to contain unnecessary transposition operators.

[0146] To this end, the present disclosure provides a computational graph optimization method that can remove unnecessary transposition operators in the computational graph, thereby reducing unnecessary resource overhead and improving the reasoning performance of the deep learning model. Figure 8a As shown, the following steps are included.

[0147] In step S801, multiple paths are obtained from the computation graph G1. Each of the multiple paths is a path extending from an input operator to an output operator of the computation graph to be optimized and including at least one transposition operator.

[0148] refer to Figure 7As shown, the computation graph G1 to be optimized can be the first computation graph 701 or the second computation graph 703, that is, the computation graph G1 to be optimized is a complete static computation graph of a deep learning model for a specific framework or a complete static computation graph after model processing. The computation graph G1 to be optimized can also be a partial computation graph (also called a subgraph) of the first computation graph 701 or a partial computation graph of the second computation graph 703. The subgraph comes from a graph slicing operation on the complete computation graph. The graph slicing operation can use any graph slicing strategy. For example, the graph is sliced using an operator that cannot be executed by the specific acceleration unit as a delimiter to obtain multiple subgraphs, and one of the subgraphs is used as the computation graph G1 to be optimized. The data format of the computation graph G1 to be optimized is determined by the data format of its input tensor. If it is a complete computation graph, the data format of the input tensor is specified in the code that generates the computation graph, and its data format is specified after the computation graph is generated. The data format of a subgraph is also the data format of its input tensors. The data format of a subgraph may be the same as that of the complete computational graph. However, since the complete computational graph includes a transpose operator, the location of the slice will affect the data format of the subgraph, so the data format of the subgraph may also differ from that of the complete computational graph. Of course, you can ensure that the data format of the subgraph and the complete computational graph remain consistent by selecting the slice location. Regarding the data format of computational graphs, currently mainstream frameworks support at least one of the two data formats: NHWC and NCHW. For example, the TensorFlow framework supports both NHWC and NCHW, so the computational graph generated by the TensorFlow framework is either NHWC or NCHW. The data format of the acceleration unit is selected during manufacturing, and currently most acceleration units select one of the two data formats mentioned above.

[0149] This step can be implemented using the following implementation methods. The first implementation method first searches for all paths that extend from the input operator to the output operator and contain at least one other transposition operator, and forms all paths into a path set; then, for each path in the path set, determines whether the transposition operator on the path is the same as the transposition operator in the remaining paths. If the two paths have the same transposition operator, the path and the path with the same transposition operator are marked as the same path, and only one of the paths is retained in the path set. This step repeatedly removes paths that do not meet the requirements from the path set to ensure that any two paths in the path set contain at least one different transposition operator. The second implementation method first constructs a blank path set, then searches for each path that extends from the input operator to the output operator, and then determines whether the transposition operator in the path is the same as the transposition operator in the existing paths in the path set. If they are the same, the path is not added to the path set, otherwise the path is added to the path set.

[0150] In step S802 , the merging direction of each transposition operator on each path is determined.

[0151] The transposition operator is used to convert a tensor from one data format to another. Therefore, this step determines the merging direction of each transposition operator based on the direction in which each transposition operator converts the format of the tensor.

[0152] Taking the data formats NHWC and NCHW as an example, the conversion from NHWC to NCHW is the first direction, and the conversion from NCHW to NHWC is the second direction. Therefore, the merging direction of the transpose operator performing the NHWC to NCHW conversion can be defined as the first direction, and the merging direction of the transpose operator performing the NCHW to NHWC conversion can be defined as the second direction. Of course, the transpose operator can also be used to complete the conversion of three or more data formats. Similarly, the merging direction of each transpose operator is determined according to the format conversion direction of the tensor.

[0153] In step S803, the transposition operators that can be offset in one or more paths are determined according to the merging direction of each transposition operator in each path, and the transposition operators that can be offset in one or more paths are cleared to obtain the optimized computation graph G2.

[0154] Take two merging directions as an example. The first direction is to convert the data format from NHWC to NCHW, and the second direction is to convert the data format from NCHW to NHWC. It can be seen that for the same tensor, the two conversions are operations that can be offset. However, for the offsettable transposition operator in step S803, it is not only necessary to determine whether the operation of the transposition operator itself can be offset according to the merging direction, but also to determine whether the clearing operation will affect the operators on other paths according to the path where the transposition operator is located. The principle of setting is that after the transposition operator is cleared, the data format of the output tensor of the calculation graph will not change. Only such a transposition operator can become an offsettable transposition operator. For this step, an optional implementation method is to construct conditions on the path and the merging direction, and check the transposition operators one by one according to the conditions to determine the offsettable transposition operators. The offsettable transposition operators should include two or more transposition operators. Give an example.

[0155] This embodiment first determines the merging direction of each transposition operator in each path. Based on the merging direction, it then determines whether the transposition operators can be offset. If offset is possible, the transposition operators that can be offset are removed. By removing unnecessary transposition operators from the computation graph, resource overhead is reduced and the inference performance of the computation graph is improved.

[0156] Figure 8b A computation graph optimization method according to another embodiment of the present disclosure is shown, which includes steps S811 to S816.

[0157] In step S11, the data format of the initial computation graph G0 is compared with the data format selected by the specific acceleration unit to see if they are the same. If they are the same, step S813 is executed; if they are different, step S812 is executed.

[0158] In step S812, a transpose operator is inserted before the input operator and after the output operator of the initial computation graph G0 to obtain the computation graph G1 to be optimized.

[0159] In step S813, the computation graph G0 is used as the computation graph G1 to be optimized.

[0160] In the above steps, it is determined whether the data format of the initial computation graph G0 is the same as the data format selected by the specific acceleration unit. If they are the same, computation graph G0 is directly used as computation graph G1. If they are different, the first transposition operator and the second transposition operator are sequentially inserted before the input operator of the initial computation graph G0 (the first transposition operator is inserted first, the second transposition operator is inserted second), and the third transposition operator and the fourth transposition operator are sequentially inserted after the output operator of the computation graph G0 (the third transposition operator is inserted first, the fourth transposition operator is inserted second), thereby obtaining the computation graph G1 to be optimized. The first and third transposition operators are used to convert the data format of the initial computation graph G0 to the data format selected by the specific acceleration unit, and the second and fourth transposition operators are used to convert the data format selected by the specific acceleration unit to the data format of the initial computation graph G0. If the initial computation graph G0 has multiple input operators, the first and second transposition operators are sequentially inserted before each input operator. If the initial computation graph G0 has multiple output operators, the third and fourth transposition operators are sequentially inserted after each output operator. It should be understood that after this process, the data format of the computation graph to be optimized G1 remains the same as that of the initial computation graph G0.

[0161] In step S814, multiple paths are obtained from the computation graph G1 to be optimized.

[0162] In step S815 , the merging direction of each transposition operator on each path is determined.

[0163] In step S816, the transposition operators that can be offset in one or more paths are determined according to the merging direction of each transposition operator in each path, and the transposition operators that can be offset in one or more paths are cleared to obtain the optimized computation graph G2.

[0164] Among them, the operations of steps S814-S816 are the same as those of steps S801-S803 and will not be repeated here. However, it should be emphasized that when the input calculation graph of step S814 is a calculation graph into which the first transposition operator to the fourth transposition operator are inserted, since the first transposition operator becomes the input operator of the calculation graph to be optimized G1 and the fourth transposition operator becomes the output operator of the calculation graph to be optimized G1, step S814 actually determines multiple paths extending from the first transposition operator to the fourth transposition operator, and each path has at least one transposition operator different from the other paths, and the path that only contains the first transposition operator to the fourth transposition operator does not belong to the path determined according to step S814.

[0165] Figure 9a and 9b yes Figure 8a The flowchart of an embodiment of the combination of steps S802 and S803 in FIG8 and the combination of steps S815 and S816 in FIG8b specifically includes the following steps.

[0166] In step S901, read each transposition operator on each path of the path set

[0167] In step S902 , it is determined whether the data format of the output tensor of the current transpose operator is the same as the data format selected by the specific acceleration unit.

[0168] In step S903 , if they are the same, it is determined that the merging direction of the current transposition operator is upward.

[0169] In step S904, if they are different, it is determined whether the data format of the input tensor of the current transpose operator is the same as the data format selected by the specific acceleration unit. If they are the same, step S905 is executed.

[0170] In step S905 , if they are the same, it is determined that the merging direction of the current transposition operator is downward.

[0171] In step S906, it is determined whether the merging directions of all transposition operators on all paths have been determined. If not, the process jumps to step S901.

[0172] In step S907 , each transposition operator of each path is traversed in sequence.

[0173] In step S908, for adjacent transposition operators a and b in the same path whose merging directions are downward and upward, respectively, it is determined whether adjacent transposition operators a and b exist only in the same identical path. If so, step S909 is executed; if not, step S910 is executed.

[0174] In step S909 , it is determined that the transposition operator a and the transposition operator b are transposition operators that can be canceled, and the transposition operator a and the transposition operator b are cleared.

[0175] In step S910, it is determined whether transposition operator a or transposition operator b exists independently in at least one path. If transposition operator a exists independently in at least one path, step S911 is executed; if transposition operator b exists independently in at least one path, step S912 is executed.

[0176] In step S911, if it is determined that the merging direction of the subsequent transposition operator adjacent to transposition operator a in all paths containing transposition operator a is upward, then transposition operator a and the subsequent transposition operator adjacent to transposition operator a are determined to be cancelable transposition operators, and a clearing operation is performed. If, on the other hand, the merging direction of the subsequent transposition operator adjacent to transposition operator a in all paths containing transposition operator a is not upward, then transposition operator a and the subsequent transposition operator adjacent to transposition operator a are non-cancellable transposition operators, and therefore no clearing operation is performed.

[0177] In step S912, if it is determined that the merging direction of the previous transposition operator adjacent to transposition operator b in all paths containing transposition operator b is downward, then it is determined that transposition operator b and the previous transposition operator adjacent to transposition operator a are cancelable transposition operators, and a clearing operation is performed. If the merging direction of the previous transposition operator adjacent to transposition operator b in all paths containing transposition operator b is not downward, it means that transposition operator b and the previous transposition operator adjacent to transposition operator b are non-cancellable transposition operators, and therefore no clearing operation is performed.

[0178] This embodiment defines the format conversion direction of the transposition operator as downward and upward, and searches for two adjacent transposition operators with the merging direction of downward and upward on each path, and judges the possible path conditions of the two transposition operators, determines the transposition operators that can be offset, and clears the transposition operators that can be offset.

[0179] This embodiment describes how to determine the merging direction of each transposition operator under two data formats and how to determine the transposition operators that can be offset according to the merging direction of each transposition operator. By clearing the transposition operators that can be offset, resource overhead is reduced and the reasoning performance of the computational graph is improved.

[0180] It should be emphasized that, although the above embodiments mostly take two data formats as examples, the embodiments of the present disclosure can also be applied to three or more data formats.

[0181] Figures 10a-10b This is an example obtained according to an embodiment of the present disclosure.

[0182] like Figure 10a As shown, 11 represents the computation graph to be optimized, which is a subgraph including path 1 consisting of conv(1), trans(2), relu(3), conv(4), tans(5) and add(6) and path 2 consisting of conv(1), trans(2), relu(3), conv(7), trans(8) and mul(9). The input is NHWC and the output is NHWC. Assuming that the data format of a specific acceleration unit is NHW C, there is no need to insert the transposition operator trans before the input operator conv(1) and after the output operators add(6) and mul(9) of the computation graph. The transposition operators trans of path 1 and path 2 are analyzed. As shown in the figure, trans(2) converts NHWC to NCHW in the downward direction, tra ns(5) converts NCHW to NHWC in the upward direction, and trans(8) converts NCHW to NH WC in the upward direction. The above example belongs to the multi-branch case. trans(2) exists in both paths 1 and 2. Therefore, when analyzing whether trans(2) can cancel out trans(5) and trans(7), we need to analyze whether trans(2) can cancel out trans(5) and trans(7) separately. According to the above embodiment, trans(5) and trans(8) on both branches of this example can cancel out trans(2). Therefore, trans(2), trans(5), and trans(8) can be deleted from computation graph 11, and the optimized computation graph is obtained.

[0183] like Figure 10bAs shown, 12 represents the computation graph to be optimized. It is a subgraph including path 1 consisting of conv(1), trans(2), trans(3), relu(4), conv(5), tans(6), and add(7), and path 2 consisting of conv(1), trans(2), trans(3), relu(4), conv(8), trans(9), and mul(10). The input is NCHW, the output output1 is NHWC, and the output output 2 is NHWC. Assuming that the data format of a specific acceleration unit is NCHW, there is no need to insert any transposition operators before the input operator conv(1) and after the output operators add(7) and mul(10) of this computation graph. The transposition operators trans of path 1 and path 2 are analyzed. As shown in the figure, t rans (2) converts NCHW to NHWC in an upward direction, trans (3) converts NHWC to N CHW in a downward direction, trans (6) converts NCHW to NHWC in an upward direction, and trans (9) converts NCHW to NHWC in an upward direction. The above example belongs to a multi-branch case. When analyzing the transposition operators on paths 1 and 2, it is necessary to start the analysis from the first downward transposition operator, i.e., trans (3), and specifically analyze whether trans (3) can be offset by trans (6) and trans (9) respectively. According to the embodiment of the present disclosure, trans (6) and t rans (9) on the two branches of this example can be offset by trans (3). Therefore, trans (3), trans (6) and trans (9) can be deleted from the calculation graph 12, and the optimized calculation graph can be obtained.

[0184] like Figure 10cAs shown, 13 represents the computation graph to be optimized. It is a subgraph consisting of path 1 consisting of conv(1), trans(2), relu(3), conv(4), and add(5), and path 2 consisting of conv(1), trans(2), relu(3), conv(6), trans(7), and mul(8). The input of computation graph 13 is NHWC, the output of output1 is NHWC, and the output of output2 is NHWC. The data format of the specific acceleration unit is NHWC, so there is no need to insert the transposition operator trans before the input operator conv(1) and after the output operators add(5) and mul(8) of this computation graph. The transposition operators of path 1 and path 2 are analyzed. As shown in the figure, trans(2) converts NHWC to NCH W in the downward direction, and trans(7) converts NCH W to NHWC in the upward direction. This example is a multi-branch case, with trans(2) existing in both paths 1 and 2. Therefore, when analyzing whether trans(2) can cancel out trans(7), we need to determine whether there are transposition operators on both paths 1 and 2 that can cancel out trans(2). However, since there is no transposition operator on path 1 that can cancel out trans(2), trans(2) cannot cancel out trans(7). In other words, computation graph 13 does not have any transposition operators that can be deleted.

[0185] In summary, the computation graph optimization method of the disclosed embodiment can obtain an optimized computation graph. The optimized computation graph reduces unnecessary transposition operators, thereby reducing resource overhead during computation graph execution.

[0186] Furthermore, although the above description uses a data center server as an example of an execution subject of the embodiments of the present disclosure, the present disclosure is not limited thereto. Theoretically, the execution subject of the embodiments of the present disclosure can be any computing device, including the servers and terminal devices described above. For terminal devices, as long as the processor, memory, and network throughput capabilities of the terminal device can meet the operating requirements of the deep learning model, the deep learning model can be deployed on it and various computational graph processing (including the computational graph processing solutions provided by the embodiments of the present disclosure) can be performed on it.

[0187] The commercial value of the disclosed embodiments

[0188] The technical solutions provided by the disclosed embodiments have practical value in integrating deep learning models with cloud computing. These embodiments can be used to optimize deep learning models that customers intend to deploy in data centers, thereby improving the inference performance of these models and, in turn, the processing efficiency of applications equipped with these models. This, in turn, increases users' willingness to host deep learning models in data centers. Therefore, these disclosed embodiments have market prospects and commercial value.

[0189] Those skilled in the art will appreciate that the present disclosure can be implemented as systems, methods, and computer program products. Therefore, the present disclosure can be specifically implemented in the following forms, namely, complete hardware, complete software (including firmware, resident software, microcode), and can also be implemented in the form of a combination of software and hardware. In addition, in some embodiments, the present disclosure can also be implemented in the form of a computer program product in one or more computer-readable media, wherein the computer-readable medium contains computer-readable program code.

[0190] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium is, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by a processing unit, device or device, or used in combination with it.

[0191] A computer-readable signal medium may include a data signal propagated in baseband or as part of a chopped wave, which carries computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction system, apparatus, or device.

[0192] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0193] The computer program code for executing the disclosed embodiments can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages, such as JAVA, C++, and can also include conventional procedural programming languages, such as C. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0194] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A processing unit, comprising: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit, configured to execute the decoded computer instructions to achieve: Determining a plurality of paths extending from an input operator to an output operator of a computation graph to be optimized and containing at least one transposition operator, wherein each of the plurality of paths has at least one transposition operator different from other paths, and at least some of the plurality of paths further include one or more of a convolution operator, an activation function operator, an addition operator, or a multiplication operator; Determine the merge direction of each transpose operator in each path; and Determining the transposition operators that can be canceled in one or more paths according to the merging direction of each transposition operator in each path, and removing the transposition operators that can be canceled in the one or more paths to obtain an optimized computation graph; The transposition operator is used to convert the data format of the tensor, and the merging direction of each transposition operator is determined according to whether the data format of the output tensor of the transposition operator is the same as the data format selected by the specific acceleration unit; wherein the acceleration unit is hardware including multiple hardware execution units; The step of determining the merging direction of each transposition operator in each path includes: For each transposition operator in each path in the path set, first determine whether the data format of the output tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is upward. If not, then continue to determine whether the data format of the input tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is downward. Then, determining the cancelable transposition operators in one or more paths according to the merging direction of each transposition operator in each path, and clearing the cancelable transposition operators in the one or more paths includes: Traversing each path in turn, for each adjacent first transposition operator and second transposition operator whose merging directions are downward and upward respectively on the path, determining whether the first transposition operator and the second transposition operator only exist in exactly the same path; if so, determining that the first transposition operator and the second transposition operator are cancelable transposition operators, and clearing the first transposition operator and the second transposition operator; If not, if it is determined that the first transposition operator exists alone in at least one path, and if it is determined that the merging direction of the subsequent transposition operator adjacent to the first transposition operator in all paths containing the first transposition operator is upward, then the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are determined to be cancelable transposition operators, and the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are cleared in all paths containing the first transposition operator; if it is determined that the second transposition operator exists alone in one path, and it is determined that the merging direction of the previous transposition operator adjacent to the second transposition operator in all paths containing the second transposition operator is downward, then the second transposition operator and the previous transposition operator adjacent to the second transposition operator are determined to be cancelable transposition operators, and the second transposition operator and the previous transposition operator adjacent to the second transposition operator are cleared in all paths containing the second transposition operator.

2. The processing unit according to claim 1, wherein: The instruction execution unit further implements: Get the initial computation graph; Determine whether the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit. If the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit, use the initial calculation graph as the calculation graph to be optimized; if the data format of the initial calculation graph is different from the data format selected by the specific acceleration unit, insert a first transposition operator and a second transposition operator in sequence before the input operator of the initial calculation graph, and insert a third transposition operator and a fourth transposition operator in sequence after the output operator of the initial calculation graph, and use the initial calculation graph after the transposition operators are inserted as the calculation graph to be optimized, wherein the first transposition operator and the third transposition operator are used to convert the data format of the calculation graph to be optimized into the data format selected by the specific acceleration unit, and the second transposition operator and the fourth transposition operator are used to convert the data format selected by the specific acceleration unit into the data format of the calculation graph to be optimized.

3. The processing unit according to claim 1, wherein: The data format of the computation graph to be optimized and the data format selected by the specific acceleration unit are one of the following data formats: NHWC and NCHW.

4. The processing unit according to any one of claims 1 to 3, wherein: The instruction execution unit further implements: slicing the complete computational graph of a specific deep learning model to obtain multiple subgraphs, and using one of the subgraphs as the computational graph to be optimized.

5. The processing unit according to any one of claims 1 to 3, wherein: The instruction execution unit also implements: converting the complete computational graph of a specific deep learning model into an intermediate expression that complies with the requirements of the specific acceleration unit, performing at least one of operator merging, quantization, and model pruning on the intermediate expression, cutting the processed intermediate expression to obtain multiple subgraphs, and using one of the subgraphs as the computational graph to be optimized.

6. The processing unit according to any one of claims 1 to 3, wherein: The computation graph to be optimized is a complete computation graph of a specific deep learning model.

7. The processing unit according to claim 1, wherein: The steps to determine multiple paths include: Searching for all paths extending from an input operator to an output operator of the computation graph to be optimized and containing at least one transposition operator, and forming all paths into a path set; For each path, it is determined one by one whether the transposition operator on the path is the same as the transposition operators on other paths. If there are different transposition operators, the path is retained in the path set.

8. The processing unit according to claim 1, wherein: The steps to determine multiple paths include: Search each path extending from the input operator to the output operator of the computation graph to be optimized, then determine whether there is a transposition operator on the path and whether the path has a different transposition operator from the existing paths in the path set. If both are true, store the path in the path set.

9. A computing device comprising an acceleration unit, a memory, and the processing unit according to any one of claims 1 to 8, wherein: The acceleration unit is hardware including a plurality of hardware execution units.

10. A computational graph optimization method for a deep learning model, comprising: Determining a plurality of paths extending from an input operator to an output operator of a computation graph to be optimized and containing at least one transposition operator, wherein each of the plurality of paths has at least one transposition operator different from other paths, and at least some of the plurality of paths further include one or more of a convolution operator, an activation function operator, an addition operator, or a multiplication operator; Determine the merge direction of each transpose operator in each path; and Determining the transposition operators that can be canceled in one or more paths according to the merging direction of each transposition operator in each path, and removing the transposition operators that can be canceled in the one or more paths to obtain an optimized computation graph; The transposition operator is used to convert the data format of the tensor, and the merging direction of each transposition operator is determined according to whether the data format of the output tensor of the transposition operator is the same as the data format selected by the specific acceleration unit; wherein the acceleration unit is hardware including multiple hardware execution units; The step of determining the merging direction of each transposition operator in each path includes: For each transposition operator in each path in the path set, first determine whether the data format of the output tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is upward. If not, then continue to determine whether the data format of the input tensor of the transposition operator is the same as the data format selected by the specific acceleration unit. If so, the merging direction of the transposition operator is downward. Then, determining the cancelable transposition operators in one or more paths according to the merging direction of each transposition operator in each path, and clearing the cancelable transposition operators in the one or more paths includes: Traversing each path in turn, for each adjacent first transposition operator and second transposition operator whose merging directions are downward and upward respectively on the path, determining whether the first transposition operator and the second transposition operator only exist in exactly the same path; if so, determining that the first transposition operator and the second transposition operator are cancelable transposition operators, and clearing the first transposition operator and the second transposition operator; If not, if it is determined that the first transposition operator exists alone in at least one path, and if it is determined that the merging direction of the subsequent transposition operator adjacent to the first transposition operator in all paths containing the first transposition operator is upward, then the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are determined to be cancelable transposition operators, and the first transposition operator and the subsequent transposition operator adjacent to the first transposition operator are cleared in all paths containing the first transposition operator; if it is determined that the second transposition operator exists alone in one path, and it is determined that the merging direction of the previous transposition operator adjacent to the second transposition operator in all paths containing the second transposition operator is downward, then the second transposition operator and the previous transposition operator adjacent to the second transposition operator are determined to be cancelable transposition operators, and the second transposition operator and the previous transposition operator adjacent to the second transposition operator are cleared in all paths containing the second transposition operator.

11. The computational graph optimization method according to claim 10, further comprising: Get the initial computation graph; Determine whether the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit. If the data format of the initial calculation graph is the same as the data format selected by the specific acceleration unit, use the initial calculation graph as the calculation graph to be optimized; if the data format of the initial calculation graph is different from the data format selected by the specific acceleration unit, insert a first transposition operator and a second transposition operator in sequence before the input operator of the initial calculation graph, and insert a third transposition operator and a fourth transposition operator in sequence after the output operator of the initial calculation graph, and use the initial calculation graph after the transposition operators are inserted as the calculation graph to be optimized, wherein the first transposition operator and the third transposition operator are used to convert the data format of the calculation graph to be optimized into the data format selected by the specific acceleration unit, and the second transposition operator and the fourth transposition operator are used to convert the data format selected by the specific acceleration unit into the data format of the calculation graph to be optimized.

12. The computational graph optimization method according to claim 10, wherein: The data format of the computation graph to be optimized and the data format selected by the specific acceleration unit are one of the following data formats: NHWC and NCHW.

13. The computation graph optimization method according to any one of claims 10 to 12, further comprising: Before the step of determining whether the data format of the computation graph to be optimized is the same as the data format selected by the specific acceleration unit, The complete computational graph of a specific deep learning model is sliced to obtain multiple subgraphs, and one of the subgraphs is used as the computational graph to be optimized.

14. The computation graph optimization method according to any one of claims 10 to 12, further comprising: Before the step of determining whether the data format of the computation graph to be optimized is the same as the data format selected by the specific acceleration unit, The complete computational graph of a specific deep learning model is converted into an intermediate expression that conforms to the specific acceleration unit, the intermediate expression is subjected to at least one of operator merging, quantization, and model pruning, the processed intermediate expression is sliced to obtain multiple subgraphs, and one of the subgraphs is used as the computational graph to be optimized.

15. The computational graph optimization method according to any one of claims 10 to 12, wherein: The computation graph to be optimized is a complete computation graph of a specific deep learning model.

16. The computational graph optimization method according to claim 10, wherein: The steps to determine multiple paths include: Searching for all paths extending from an input operator to an output operator of the computation graph to be optimized and containing at least one transposition operator, and forming all paths into a path set; For each path, it is determined one by one whether the transposition operator on the path is the same as the transposition operators on other paths. If there are different transposition operators, the path is retained in the path set.

17. The computational graph optimization method according to claim 10, wherein: The steps to determine multiple paths include: Search each path extending from the input operator to the output operator of the computation graph to be optimized, then determine whether there is a transposition operator on the path and whether the path has a different transposition operator from the existing paths in the path set. If both are true, store the path in the path set.

18. A data center comprising the computing device according to claim 9.

Images