Processing unit, computing device, and computational graph processing method for deep learning model

Abstract

Disclosed are a processing unit, a computing device, and a method for processing a computational graph of a deep learning model. The processing unit includes: an instruction fetch unit for retrieving computer instructions from a memory outside the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; an instruction execution unit for executing the decoded computer instructions to achieve: extracting the attributes of operators from a computational graph of a deep learning model used in a first deep learning framework; converting the computational graph into an intermediate expression that complies with an acceleration unit, wherein the attributes of the operators in the intermediate expression are different from the attributes of the operators in the computational graph; performing model processing on the intermediate expression; and converting the processed intermediate expression back into the computational graph of the first deep learning framework, wherein, for operators whose operator identifiers have not changed, the attributes of the operator in the intermediate expression are replaced with the extracted attributes of the operator. Compared with the prior art, the embodiment of the present disclosure requires fewer mapping functions to be maintained.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more specifically, to a processing unit, a computing device, and a computational graph processing method for a deep learning model. Background Art

[0002] Currently, mainstream deep learning frameworks include TensorFlow, MxNet, Caffe, and MxNet. These frameworks define different sets of operators, and even operators with similar functionality have different properties across different frameworks. Operators, such as the convolution operator, are the fundamental units of computation in deep learning models. In TensorFlow, the convolution operator's padding attribute is either "VALID" or "SAME," indicating that after the convolution operation, only valid convolution is performed, with no processing of boundary data, while "SAME" indicates that the convolution result at the boundary is retained. However, in MxNet and Caffe, the convolution operator's padding attribute is a two-dimensional array, allowing users to specify horizontal and vertical padding values.

[0003] Those skilled in the art are aware that for a deep learning model to run on an acceleration unit, the corresponding processing unit must perform some model processing, which is then compiled into a deep learning model supported by the acceleration unit's instruction set. This model processing includes operator merging and quantization. Operator merging is the process of fusing operators within a deep learning model, for example, combining multiple operators into a single operator. Quantization is the process of converting parameters such as weights in a deep learning model, as well as inputs to the deep learning model, from high-precision data types to low-precision data types. These model processing operations must be performed within a framework or instruction set supported by the acceleration unit. These only support the attributes and attribute values of each operator within the acceleration unit's framework, not the attributes and attribute values of each operator within the original deep learning framework of the deep learning model. Therefore, the deep learning model within the original deep learning framework must be converted into an intermediate representation supported by the acceleration unit before model processing within the framework supported by the acceleration unit can be performed. The intermediate representation after model processing is then converted back to the original framework.

[0004] Currently, converting a deep learning model from the original framework into an intermediate representation supported by the acceleration unit, and converting the processed intermediate representation back to the original framework, requires the implementation of operator mapping functions. Consequently, it is necessary to define numerous operator mapping functions to complete the computational graph conversion between the original framework and the intermediate representation, and vice versa. This approach has the disadvantage of requiring the development and maintenance of numerous mapping functions. Summary of the Invention

[0005] Based on this, the purpose of the present disclosure is to provide a processing unit, a computing device and a computational graph processing method for a deep learning model to solve the problems existing in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a processing unit, including:

[0007] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0008] an instruction decoding unit for decoding the retrieved computer instructions;

[0009] An instruction execution unit, configured to execute the decoded computer instructions to achieve:

[0010] Extracting properties of operators from a computational graph of a deep learning model used in a first deep learning framework;

[0011] Converting the computation graph into an intermediate expression conforming to the acceleration unit, wherein the attributes of operators in the intermediate expression are different from the attributes of operators in the computation graph;

[0012] performing model processing on the intermediate expression;

[0013] The processed intermediate expression is converted back into a computational graph of a deep learning model for the first deep learning framework, wherein, for operators whose operator identifiers have not changed, the attributes of the operator in the intermediate expression are replaced by the extracted attributes of the operator.

[0014] Optionally, the model processing includes at least one of the following: operator merging, model pruning, quantization, and graph slicing.

[0015] Optionally, extracting attributes of operators from a computation graph of a deep learning model used in the first deep learning framework includes:

[0016] Extracting an operator identifier, an attribute identifier, and an attribute value of an operator from a computational graph of a deep learning model under the first deep learning framework;

[0017] The extracted attribute identifiers and attribute values are stored corresponding to the operator identifiers.

[0018] Optionally, storing the extracted attribute identifier and attribute value in correspondence with the operator identifier includes: storing the operator identifier of the operator, the attribute identifier and the attribute value in the operator as a character string;

[0019] The method of replacing the attributes of the operator in the intermediate expression with the extracted attributes of the operator includes: according to the operator identifier of the operator, obtaining the attribute identifier and attribute value of the operator in the character string where the operator identifier of the operator is located, and replacing the attribute identifier and attribute value of the operator in the intermediate expression.

[0020] Optionally, a first separator is provided between character strings of different operators, and a second separator is provided between the operator identifier and the attribute identifier, and between the attribute identifier and the attribute value in the character string of the same operator.

[0021] Optionally, the instruction execution unit further implements: storing the string as a multi-level variable container, wherein a first-level container in the multi-level variable container contains an operator identifier of an operator; a second-level container is extended from the first-level container to contain an attribute identifier of the operator; a third-level container is extended from the second-level container to contain an attribute value of the attribute identifier of the operator;

[0022] The replacing the attributes of the operator in the intermediate expression with the extracted attributes of the operator includes: according to the operator identifier of the operator, obtaining the attribute identifier and attribute value of the operator contained in the second-level container and the third-level container respectively extended from the first-level container where the operator identifier of the operator is located, and replacing the attribute identifier and attribute value of the operator in the intermediate expression.

[0023] Optionally, the instruction execution unit further implements: before the step of extracting the attributes of the operator from the computation graph of the deep learning model under the first deep learning framework, sorting the computation graph and removing duplicate operators.

[0024] Optionally, converting the computation graph into an intermediate expression conforming to the acceleration unit includes: converting the attributes of at least one operator of the computation graph into attributes of a corresponding operator defined in the intermediate expression through a mapping function.

[0025] Optionally, an operator mapping table is used to organize the mapping function.

[0026] Optionally, the properties of the extracted operators are stored in intermediate representations before and after processing.

[0027] Optionally, the instruction execution unit further implements: when converting the processed intermediate expression back into a computational graph of the deep learning model for the first deep learning framework, identifying and marking operators whose operator identifiers have changed.

[0028] In a second aspect, an embodiment of the present disclosure provides a computing device comprising a memory and any one of the processing units described above.

[0029] In a third aspect, an embodiment of the present disclosure provides a method for processing a computational graph of a deep learning model, including:

[0030] Extracting properties of operators from a computational graph of a deep learning model used in a first deep learning framework;

[0031] Converting the computation graph into an intermediate expression conforming to the acceleration unit, wherein the attributes of operators in the intermediate expression are different from the attributes of operators in the computation graph;

[0032] performing model processing on the intermediate expression;

[0033] The processed intermediate expression is converted back into a computational graph of a deep learning model for the first deep learning framework, wherein, for operators whose operator identifiers have not changed, the attributes of the operator in the intermediate expression are replaced by the extracted attributes of the operator.

[0034] Optionally, the model processing includes at least one of the following: operator merging, model pruning, quantization, and graph slicing.

[0035] Optionally, extracting attributes of operators from a computation graph of a deep learning model used in the first deep learning framework includes:

[0036] Extracting operator identifiers, attribute identifiers, and attribute values of operators from a computational graph of a deep learning model used in the first deep learning framework;

[0037] The extracted attribute identifiers and attribute values are stored corresponding to the operator identifiers.

[0038] Optionally, storing the extracted attribute identifier and attribute value in correspondence with the operator identifier includes: storing the operator identifier of the operator, the attribute identifier and the attribute value in the operator as a character string;

[0039] The replacing the attributes of the operator in the intermediate expression with the extracted attributes of the operator includes: according to the operator identifier of the operator, obtaining the attribute identifier and attribute value of the operator in the character string where the operator identifier of the operator is located, and replacing the attribute identifier and attribute value of the operator in the intermediate expression.

[0040] Optionally, a first separator is provided between character strings of different operators, and a second separator is provided between the operator identifier and the attribute identifier, and between the attribute identifier and the attribute value in the character string of the same operator.

[0041] Optionally, the method further includes: before the step of extracting the attributes of the operators from the computation graph of the deep learning model under the first deep learning framework, sorting the computation graph and removing duplicate operators.

[0042] Optionally, converting the computation graph into an intermediate expression conforming to the acceleration unit includes: converting the attributes of at least one operator of the computation graph into attributes of a corresponding operator defined in the intermediate expression through a mapping function.

[0043] Optionally, the properties of the extracted operators are stored in intermediate representations before and after processing.

[0044] In a fourth aspect, an embodiment of the present disclosure provides a data center, comprising the above-mentioned computing device.

[0045] The computational graph processing method provided by the embodiments of the present disclosure extracts and stores the attribute data of each operator in the computational graph of a specific deep learning framework, and then uses the stored attribute data to restore the attributes of the corresponding operator in the intermediate expression. The prior art uses a mapping function to restore the attributes of the corresponding operator in the intermediate expression. Therefore, compared with the prior art, the embodiments of the present disclosure require developers to maintain fewer mapping functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0047] Figure 1 A hierarchical structure diagram of a data center to which an embodiment of the present disclosure is applied is shown;

[0048] Figure 2 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0049] Figure 3 This is a block diagram of the internal structure of a server in a data center according to an embodiment of the present disclosure;

[0050] Figure 4 is a control relationship diagram of a central processing unit (CPU) and an NPU inside a server according to an embodiment of the present disclosure;

[0051] Figure 5 is a diagram of the internal structure of an NPU core according to one embodiment of the present disclosure;

[0052] Figure 6 It is a layered software architecture diagram;

[0053] Figure 7 is an example graph of computational graph transformation;

[0054] Figure 8 is a flowchart of a computational graph processing method provided by an embodiment of the present disclosure;

[0055] Figure 9 is a flowchart of a computation graph processing method provided by another embodiment of the present disclosure;

[0056] Figure 10 It is a structural diagram of a multi-layer variable container. DETAILED DESCRIPTION

[0057] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0058] The following terms are used in this document.

[0059] Accelerator unit: A processing unit designed to improve data processing speed in specialized areas (e.g., image processing, neural network operations, etc.) where general-purpose processors are inefficient. It is often used in conjunction with a general-purpose processor (CPU), controlled by the general-purpose processor, to perform processing for specific purposes or fields, thereby improving computer processing efficiency in these areas. It can also be called an AI processing unit and can include a graphics processing unit (GPU), central processing unit (CPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and dedicated AI acceleration hardware (e.g., NPU).

[0060] On-chip memory: Memory that is used exclusively within the main core or sub-core and cannot be shared.

[0061] The command processor serves as the command interface between the accelerator unit and the central processing unit (CPU), which drives it. The command processor receives instructions from the CPU for the accelerator unit and distributes them to the various cores within the accelerator unit for execution. It also manages the synchronization of the cores within the accelerator unit.

[0062] Lifecycle: An operand isn't always involved in the entire instruction sequence. The period between its first appearance and its last use in the instruction sequence is the lifecycle of the operand. In other words, after its lifecycle, it's no longer needed and doesn't need to be stored in on-chip memory.

[0063] Neural network: Generally refers to an artificial neural network (ANN), an algorithmic network that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. A classic neural network, also the simplest neural network structure, consists of three layers: input layer, output layer, and middle layer (also called hidden layer). The input layer, output layer, and middle layer each contain multiple nodes.

[0064] Neural network model: In a neural network, nodes are mathematized to generate mathematical models of nodes. The mathematical models of a large number of nodes in a neural network constitute the neural network model.

[0065] Deep learning model: The concept of deep learning originates from the study of neural networks. A neural network with multiple intermediate layers is called a deep learning network. Therefore, in this sense, a deep learning model is also a type of neural network model. Both deep learning models and neural network models must be trained. Sample data is input into a designed network structure (i.e., the network structure has been determined), feature information is extracted through multiple intermediate layers, and the weight parameters of the neurons are continuously modified based on the output results of the output layer, so that the output results of the output layer increasingly converge to the preset results until the final weight parameters are determined. The trained deep learning model can be truly applied in real-world scenarios. At the same time, the use of the deep learning model in real-world scenarios can also be collected to optimize the deep learning model.

[0066] Node: The smallest unit of independent operation in a deep learning model, which receives input and generates output after operating on its own weight parameters or parameters in other models (such as hyperparameters). A deep learning model can include various specific operations such as convolution and pooling, and has various operation nodes including convolution nodes and pooling nodes. There are multiple layers in a deep learning model, each layer has multiple nodes, and the output of each node is the input of the node in the next layer. Specifically, a node includes a program for a specific operation and related data. For example, a convolution operation node includes the program code used for the convolution operation and some data used in the convolution.

[0067] An operator is a collection of operations built into a deep learning model to implement specific functionality. Each layer of a deep learning model can contain multiple such operators. These are referred to as operations in the TensorFlow framework and layers in the Caffe framework. Operators are considered a further abstraction based on nodes; an operator can correspond to one or more nodes. Therefore, operators and nodes can sometimes represent the same program code.

[0068] Instruction set: A collection of instructions for operations supported within the chip, such as operations that mainly support deep learning operators, such as Convolution, Pooling, ROI, etc.

[0069] Quantization: The act of converting the inputs of operation nodes in a deep learning model, as well as the weight parameters and other parameters of the operation nodes, from high-precision data types to low-precision data types, thereby reducing the requirements for data throughput and storage space.

[0070] Dequantization: The opposite process of quantization, which is the process of converting the input of the operation node in the deep learning model, as well as the weight parameters and other parameters of the operation node from low-precision data types to high-precision data types.

[0071] Intermediate Representation (IR): Deep learning models have different formats, depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet. The code representations of these deep learning models also differ. This poses significant challenges to the universality of deep learning model quantization. An intermediate representation (IR) converts the various deep learning model code formats into a single representation consistent with one or more acceleration units. It analyzes the meaning of each code statement in a deep learning model and translates it into a common representation based on its meaning. This ensures that code statements with the same meaning across different deep learning models are expressed identically in the IR. Currently, there are tools available that convert the representations of different deep learning models into IR.

[0072] Computation graph: Currently, deep learning frameworks mainly have two programming methods: declarative programming and imperative programming. In declarative programming, the program code first defines a neural network model structure to describe the calculation logic, but it is not executed immediately. The neural network model structure is only executed when the program code that calls the network model structure is executed. The neural network model structure includes multiple operators (or symbolic expressions of operators) and the connection relationship between them, and can be shown graphically. Therefore, the neural network model structure is called a static computation graph. In imperative programming, the program code directly returns the result of the operation, and the definition and execution of the neural network model structure are synchronized. Generally speaking, static graphs are convenient for compiling and optimizing the overall neural network model, which is more conducive to performance improvement, while dynamic graphs are very convenient for users to debug specific programs.

[0073] Figure 1 A hierarchical structure diagram of a data center is shown as a scenario in which an embodiment of the present disclosure is applied.

[0074] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Neural networks, a key AI technology, are already widely used in data center big data analytics.

[0075] In traditional large data centers, the network structure is usually Figure 1The three-layer structure shown is a hierarchical inter-networking model. This model consists of the following three layers:

[0076] Access Layer 103: Sometimes also called the edge layer, it includes access switches 130 and the servers 140 connected to them. Servers 140 are the processing and storage entities of the data center, processing and storing large amounts of data within the data center. Access switches 130 connect these servers to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, hence the name "top of rack" switches, physically connecting the servers.

[0077] Aggregation Layer 102: sometimes also called the distribution layer, includes aggregation switches 120. Each aggregation switch 120 connects to multiple access switches and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0078] Core Layer 101: Includes core switch 110. Core switch 110 provides high-speed forwarding for packets entering and exiting the data center, and provides connectivity for multiple aggregation layers. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0079] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0080] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with other aggregation switches 120 being used only in the event of a failure (dashed lines in the figure above). This means that horizontal scalability is impossible at the aggregation layer because even if multiple aggregation switches 120 are added, only one is still operational.

[0081] Figure 2 Shown Figure 1 The physical connection of each component in the hierarchical data center. Figure 2 As shown, a core switch 110 is connected to multiple aggregation switches 120 , an aggregation switch 120 is connected to multiple access switches 130 , and an access switch 130 is connected to multiple servers 140 .

[0082] server

[0083] Since server 140 is the real device in the data center, Figure 3 The following figure shows a block diagram of the internal structure of a server 140. Server 140 includes a bus-connected memory 210, a central processing unit (CPU) 220, and various acceleration units. These acceleration units include an embedded neural network processor (NPU) 230, a data transfer unit (DTU) 260, a graphics processing unit (GPU, not shown), an application-specific integrated circuit (ASIC, not shown), and a field-programmable gate array (FPGA, not shown).

[0084] The traditional processor architecture design results in control units and storage units occupying a large portion of the space, while the computing unit occupies less space. As a result, while highly effective for logic control, it is inefficient for large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively increase the speed of computation for different functions and fields. The acceleration unit proposed in this disclosure can be any of these types, and each of these acceleration units is described below.

[0085] NPU230: It is a processing unit that uses a data-driven parallel computing architecture to handle a large number of operations (such as convolution, pooling, etc.) of each neural network node. Since the data and intermediate results of a large number of operations (such as convolution, pooling, etc.) of each neural network node are closely related throughout the entire calculation process and will be used frequently, with the existing CPU architecture, since the memory capacity within the CPU core is very small, a large number of external core memories must be accessed frequently, resulting in inefficient processing. With the NPU, each core has on-chip memory with a storage capacity suitable for neural network calculations, avoiding frequent access to memory outside the core, which can greatly improve processing efficiency and computing performance.

[0086] Data Transfer Unit (DTU) 260: This is a wireless terminal device specifically designed to convert serial port data into IP data or vice versa for transmission over a wireless communication network. The DTU's primary function is to wirelessly transmit data from remote devices back to the backend center. At the front end, the DTU and the client's device are connected via an interface. After powering on, the DTU first registers with the mobile GPRS network and then establishes a socket connection with the backend center installed in the DTU. The backend center acts as the server of the socket connection, and the DTU is the client of the socket connection. Therefore, the DTU is used in conjunction with the backend software. Once the connection is established, the front-end device and the backend center can perform wireless data transmission via the DTU.

[0087] A graphics processing unit (GPU) is a microprocessor specifically designed for image and graphics-related computing. The GPU addresses the CPU's lack of computing space by utilizing a large number of dedicated graphics processing units. This reduces the graphics card's reliance on the CPU and allows it to handle some of the computationally intensive image processing tasks previously handled by the CPU.

[0088] Application-Specific Integrated Circuit (ASIC): An integrated circuit designed and manufactured to meet specific user requirements and the needs of a specific electronic system. Because this type of integrated circuit is customized according to user requirements, its structure is often adapted to that specific user's needs.

[0089] Field Programmable Gate Array (FPGA): A further development of programmable devices such as PALs and GALs. It emerged as a semi-custom circuit within the field of Application-Specific Integrated Circuits (ASICs), addressing both the shortcomings of custom circuits and the limited number of gates inherent in existing programmable devices.

[0090] Although the acceleration unit has the advantage of being much more efficient than ordinary processors for specific applications or fields, it is also subject to the control of the processing unit 220. Taking the NPU as an example, the memory 210 stores various deep learning models, including the neurons and neuron weight data of these models. These deep learning models are called when needed. Figure 3One processing unit 220 is deployed to an NPU 230. Specifically, the processing unit 220 can inform the NPU 230 of the storage location of the NPU 230 deep learning model in the memory 210 in the form of instructions. The NPU 230 can then address these locations and store the instructions to be executed in its on-chip memory. The processing unit 220 can also send instructions to be executed by the NPU 230 in the form of instructions. The NPU 230 receives the instructions and stores them in the on-chip memory. Similarly, the NPU 230 can also obtain input data in the above manner. The NPU 230 obtains the instructions to be executed and the input data to perform inference calculations. The weight parameters of the node can be included in the instruction sequence of the deep learning model and retrieved from the memory 210 by the NPU 230. Of course, the weight parameters of the node can also be stored independently and retrieved from the memory 210 by the NPU 230 when needed. Here, the processing unit 220 can be understood as a hardware unit with scheduling and control capabilities, generally a hardware unit such as a central processing unit (CPU), a microcontroller, or a microprocessor.

[0091] Internal Structure of the Processing Unit and NPU 230

[0092] The following combination Figure 4 The internal structure diagram of the processing unit and NPU230 illustrates how the processing unit controls the operation of the acceleration unit.

[0093] like Figure 4 As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0094] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 4 The register file 229 shown is used to store a register for instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0095] After fetching the instruction, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0096] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0097] After the instruction issuing unit 225 sends the instruction to the instruction executing unit 226, the instruction executing unit 226 begins executing the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is a neural network inference instruction, the instruction executing unit 226 will no longer execute the instruction, but instead send the instruction to the NPU 230 via the bus for execution.

[0098] NPU 230 includes multiple cores 236 ( Figure 4 4 cores are shown in the figure, but those skilled in the art should understand that NPU230 may also include other numbers of cores 236), command processor 237, direct storage access mechanism 235, and bus channel 231.

[0099] The bus channel 231 is a channel for instructions to enter and exit the NPU 230 from the bus. According to different mechanisms, the bus channel 231 may include a PCIE channel 232, an I2C channel 233, and a JTAG channel 234.

[0100] PCIE, or PCI-Express, is a high-speed serial computer expansion bus standard proposed by Intel in 2001 to replace the old PCI, PCI-X, and AGP bus standards. PCIE is a high-speed serial point-to-point dual-channel high-bandwidth transmission. The connected devices are allocated exclusive channel bandwidth and do not share bus bandwidth. It mainly supports active power management, error reporting, end-to-end reliable transmission, hot plugging, and quality of service functions. Its main advantage is its high data transmission rate, and it also has considerable development potential. At present, most PCIE buses are PCIE GEN3, but the disclosed embodiments can also use PCIE GEN4, that is, a bus channel that complies with the PCI-Express 4.0 standard.

[0101] I2C channel 233 is a simple, bidirectional, two-wire synchronous serial bus developed by Philips. It only requires two wires to transmit information between devices connected to the bus.

[0102] JTAG stands for Joint Test Action Group and is the common name for IEEE Standard 1149.1, Standard Test Access Port and Boundary-Scan Architecture. This standard is used to verify the functionality of designed and manufactured printed circuit boards. JTAG was formally standardized in 1990 by IEEE Document 1149.1-1990. In 1994, a supplementary document describing the Boundary-Scan Description Language (BSDL) was added. Since then, this standard has been widely adopted by electronics companies worldwide. Boundary scan has become almost synonymous with JTAG. JTAG Channel 234 is a bus channel that complies with this standard.

[0103] The direct memory access (DMA) mechanism 235 is a function provided by some computer bus architectures that enables data to be written directly from an attached device (such as an external memory) to the on-chip memory of the NPU 230. Compared to the method in which all data transmission between devices must pass through the command processor 237, this method greatly improves the efficiency of data access. It is precisely because of this mechanism that the core of the NPU 230 can directly access the memory 210, read the parameters in the deep learning model (such as the weight parameters of each node), etc., which greatly improves the efficiency of data access. Although the direct memory access mechanism 235 is shown in the figure as being located between the processor 237 and the bus channel 231, the design of the NPU 230 is not limited to this. In some hardware designs, each NPU core 236 can include a direct memory access mechanism 235, so that the NPU core 236 does not need to go through the command processor 237, but directly reads data from the attached device and writes it to the on-chip memory of the NPU 230.

[0104] The command processor 237 assigns the instructions sent by the processing unit 220 to the NPU 230 to the core 236 for execution. The instruction execution unit 226 sends the pending instructions that need to be executed by the NPU 230 to the NPU 230, or the instruction execution unit 226 informs the storage location of the pending instructions on the memory 210. After the pending instruction sequence enters from the bus channel 231, it is cached in the command processor 237, and the command processor 237 selects the core 236 and assigns the instruction sequence to it for execution. The pending instructions come from the compiled deep learning model. It should be understood that the pending instruction sequence may include pending instructions executed by the processing unit 220 and pending instructions that need to be executed by the NPU 230.

[0105] NPU core

[0106] Figure 5 FIG. 4 is a diagram of the internal structure of an NPU core according to an embodiment of the present disclosure.

[0107] In one embodiment, Figure 5 As shown, the NPU core 236 includes a tensor engine 310, a pooling engine 320, a convolution process 330, an activation operation 380, a sequencer 350, an instruction cache 340, an on-chip memory 360, and a constant buffer 370. Among them, the tensor engine 310, the pooling engine 320, the convolution process 330, and the activation operation 380 are all hardware execution units. The hardware execution unit is the hardware module that actually performs various operations. Some other hardware execution units are not shown in the figure.

[0108] The instruction sequence assigned by the command processor 237 to the NPU core 236 first enters the instruction cache 340 cache. Then, the sequencer 350 fetches instructions from the instruction cache 340 in a first-in, first-out order and assigns them to various hardware execution units for execution according to the nature of the instructions. The tensor engine 310 is responsible for processing tensor-related operations in the deep learning model. The pooling engine 320 is responsible for processing pooling operations in the deep learning model. The convolution processing 330 is responsible for the convolution operation in the deep learning model. The activation operation 380 is used to perform the operation corresponding to the activation function in the deep learning model. The sequencer 350 decides to assign the instruction to each hardware execution unit for execution based on the nature of the operation, such as convolution, matrix multiplication, or pooling, taken out.

[0109] The on-chip memory 360 is an in-core memory that stores the weight parameters in the deep learning model, as well as the inputs and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning model (for example, hyperparameters in the deep learning model) in addition to the weight parameters. As described above, in the process of the processing unit 220 pre-configuring the deep learning model in the NPU 230, the processing unit 220 can send the location of the parameters in the model in the memory 210 to the NPU 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the NPU 230 will take them out from the corresponding location of the memory 210 when needed and place them in the on-chip memory 360. For other parameters, the NPU 230 will take them out from the corresponding location of the memory 210 when needed and place them in the constant buffer 370. In addition, when the executable instruction is assigned to the core 236 for execution by the command processor 237, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0110] Software architecture diagram

[0111] Improvements to deep learning models require not only the aforementioned hardware support but also continuous improvements at the software and algorithm levels. Only by combining underlying hardware support with the deep learning algorithm architecture above can a powerful computing engine be delivered.

[0112] Figure 6 This is a diagram of a layered software architecture. Layered software design is the mainstream design approach for large-scale software projects. It reduces dependencies between layers, allowing developers to focus on a single layer within the overall structure and easily replace existing layer implementations with new code.

[0113] As shown in the figure, from top to bottom, the software architecture diagram includes an application layer 401, a framework layer 402 and a function layer.

[0114] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, and recommendation 407. These applications are built using this architecture and can also call the architecture to provide runtime interfaces to gain reasoning capabilities within the application.

[0115] The framework layer 402 integrates various deep learning frameworks, such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to enable continued optimization and improvement of various algorithms. TensorFlow 408 is a symbolic mathematics system based on data flow programming and is widely used in programming various machine learning algorithms. MXNet 409 is the deep learning library of choice for Amazon. Caffe 410, or Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modularity.

[0116] The functional layer includes a compilation stack 403 and a runtime stack 404. The compilation stack 403 is used for various model conversions (converter 411), quantization 412, optimization 413, and compilation 414. Conversion 411 converts the model's internal data into an intermediate representation (IR) format. Quantization 412 converts parameters such as weights in a deep learning model, as well as the inputs to the deep learning model, from high-precision data types to low-precision data types. Optimization 413 fuses operators within the model and performs operations such as multi-model optimization linking. Compilation 414 optimizes the model based on the hardware, generating a binary model that the hardware can recognize. The runtime stack 404 includes a runtime API 415, an execution manager 416, a user-mode driver 417, and a kernel-mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized runtime API 415 provides various runtime callable interfaces. The user-mode driver 417 provides kernel-mode hardware commands and resource scheduling. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0117] By integrating multiple mainstream deep learning models into a single open source platform, developers can develop, compile, and run multiple deep learning models on the same platform, eliminating the need to deploy and maintain multiple model frameworks. Furthermore, the open source platform can be expanded to support more deep learning models.

[0118] Computational Graph Conversion

[0119] Computational graph conversion converts a computational graph from one standard to another. These differences in standards mean that different deep learning frameworks define different representations for computational graphs. Deep learning frameworks are integrated environments that support the compilation and execution of computational graphs. When a computational graph from one framework needs to be processed in another framework, computational graph conversion is required.

[0120] In particular, a large number of accelerator units with different instruction set architectures have emerged. To deploy the computation graph of a specific deep learning framework on an accelerator unit with a specific instruction set architecture, the computation graph must be adapted and optimized for that accelerator unit. This has led to the development of intermediate representations. Intermediate representations are computation graphs defined based on the instruction set architecture of one or more accelerator units. Computation graphs from different frameworks are first converted into intermediate representations, and then developers focus on performing various model processing on the intermediate representations. At the same time, after model processing, the intermediate representations sometimes need to be returned to the original framework for training and model improvement. Therefore, the intermediate representations, after model processing, need to undergo further computational graph conversion and be converted back to the original framework. This process can be summarized as: converting the computation graph of the original framework into the intermediate representation, processing the intermediate representation, and then converting the processed intermediate representation back to the computation graph of the original framework. This process can be repeated.

[0121] The following describes computation graph conversion based on the aforementioned software architecture diagram. The framework layer 402 can provide computation graphs for deep learning models in various frameworks to the compilation stack 403 or application layer 401. After receiving the computation graph, the compilation stack 403 converts it into an intermediate representation and then performs model processing on the intermediate representation. The processed intermediate representation can then be deployed to a designated acceleration unit via the runtime stack 404. During this process, all generated computation graphs are static.

[0122] Continue to refer Figure 7 As shown. First, as shown in the figure, various deep learning frameworks support models A to M, and the first computational graph 701 of a specific deep learning framework is converted into an intermediate expression 702. The intermediate expression 702 predetermines many operators and their attributes. Converting the first computational graph 701 into the intermediate expression 702 includes: each operator and its attributes in the first computational graph 701 are converted into the corresponding operator and its attributes defined by the intermediate expression 702. This conversion can be achieved with the help of conversion 411. Conversion 411 defines a mapping function to achieve operator conversion. The mapping function is used to convert between operators with the same function but different attributes. Developers understand the specific functions of each operator and their respective attribute definitions, and predefine mapping functions for operators with the same function but different attributes. Referring to the operator mapping table shown in Table 1, the left column is the operator identifier of the first operator, and the right column is the name of the mapping function.

[0123] Table 1

[0124]

[0125] Of course, the embodiments of the present disclosure are not limited to using a mapping function to perform operator attribute conversion, and other methods may also be used to complete operator attribute conversion.

[0126] Then, the intermediate expression 702 is subjected to model processing such as operator merging, model pruning, quantization, and graph slicing, and the second computational graph 703 is output. Quantization processing inserts quantization nodes and dequantization nodes into the intermediate expression 702. Operator merging merges two or more operators into one operator based on the hardware. Graph slicing is to divide the intermediate expression into several subgraphs to facilitate reading and processing by the acceleration unit. Model pruning is a model compression method that introduces sparsity to the dense connections of the deep learning model and reduces the number of non-zero weights by directly setting "unimportant" weights to zero. The second computational graph 703 can be deployed to a designated acceleration unit for execution.

[0127] Finally, if necessary, the second computation graph 703 is converted back to a computation graph supported by the specific deep learning framework. Specifically, this includes converting the operators and their attributes in the second computation graph 703 into operators and their attributes supported by the specific deep learning framework. This conversion can also be achieved using a mapping function. Of course, this step can also be implemented using other methods, such as the method used in the computation graph processing method of the embodiment of the present disclosure, which will be described below.

[0128] In addition, since the computation graph contains various operators and the connection relationships between them, after the operator conversion, it is also necessary to ensure that the connection relationships between the operators are correct, or the connection relationships between the operators can be directly reconstructed.

[0129] The computational graph processing method of the embodiment of the present disclosure

[0130] The embodiments of the present disclosure provide a computational graph processing method for a deep learning model, which is used to implement computational graph conversion. According to the computational graph processing method, developers need to maintain fewer mapping functions than in the prior art. Figure 8 It is a flowchart of the computational graph processing method, which specifically includes the following steps.

[0131] In step S801 , attributes of operators are extracted from a first computation graph of a specific deep learning framework.

[0132] A specific deep learning framework is a framework used for integrated development and operation of a certain deep learning model, such as the TensorFlow framework, MxNet framework, Caffe framework, MxNet framework, etc.

[0133] The first computational graph consists of multiple operators (or symbolic expressions representing operators) defined by a specific deep learning framework, and the connections between them. The connections between operators are handled in a pointer-like manner. For example, in a computational graph, if operators A and B point to operator C using a pointer-like method, this means that the outputs of operators A and B are used as the inputs of operator C, and the connection between A and B is now established. The connection between operators is not the focus of this article and will not be described in detail.

[0134] This step is used to obtain the properties of operators in the first computation graph. It includes extracting and reading each operator from the first computation graph one by one, and then extracting the properties of each operator. Each operator's properties include an operator identifier, an attribute identifier, and an attribute value. Since each operator can have multiple attributes, the properties of each operator may include: an operator identifier, multiple attribute identifiers, and the attribute value corresponding to each attribute identifier. The extracted attribute identifiers and attribute values are stored in any storage unit corresponding to the operator identifier for use in subsequent steps.

[0135] In step S802, the first computation graph is converted into an intermediate representation.

[0136] The operators of the first computation graph are operators that have been registered under a specific framework. The intermediate expression defines multiple operators according to the instruction set architecture of the specified acceleration unit. The conversion between the two can be achieved through a mapping function. The mapping function can be referred to as shown in Table 1. When the mapping function is used to map at least one operator of the first computation graph to the corresponding operator defined by the intermediate expression, the operator attribute conversion is completed, and the intermediate expression corresponding to the first computation graph is obtained. Combined Figure 7 , this step is to convert the first computation graph 701 in the figure into an intermediate expression 702.

[0137] In step S803, the intermediate expression is model-processed to obtain a second computation graph.

[0138] refer to Figure 7 As shown, the intermediate expression 702 is subjected to model processing such as operator merging, model pruning, quantization, and graph slicing, and a second computational graph 703 is output. Quantization processing inserts quantization nodes and dequantization nodes into the intermediate expression 702. Operator merging merges two or more operators into one operator based on the hardware. Graph slicing is to divide the intermediate expression into several subgraphs to facilitate reading and processing by the acceleration unit. Model pruning is a model compression method that introduces sparsity to the dense connections of the deep learning model and reduces the number of non-zero weights by directly setting "unimportant" weights to zero. The second computational graph 703 can be deployed to a designated acceleration unit for execution.

[0139] In step S804, the second computation graph is converted into a computation graph of a specific deep learning framework. For operators whose operator identifiers have not changed, the attributes of the operators extracted in step S801 are used to replace the attributes of the operators in the intermediate representation.

[0140] Specifically, the previously stored operator identifiers are used one by one to retrieve the second computation graph. For the same operator identifiers retrieved, the attribute identifier and attribute value of the operator previously stored are compared with the attribute identifier and attribute value of the corresponding operator in the second computation graph. If there are differences, the attribute identifier and attribute value of the operator previously stored are used to replace the current attribute identifier and attribute value of the operator in the second computation graph.

[0141] The third computational graph thus obtained has attribute identifiers and corresponding attribute values unique to the original deep learning framework, so it can be provided to a specific deep learning framework so that developers can continue forward processing of the specific deep learning framework.

[0142] The disclosed embodiments are applicable to computational graph conversion operations for deep learning models in multiple frameworks. First, the attributes of one or more operators in the first computational graph of a specific deep learning framework are extracted and stored. The attributes are then used to restore the current attributes of the corresponding operators in the intermediate representation to the previously stored attributes. Compared to the prior art, which uses mapping functions to restore the attributes of the corresponding operators in the intermediate representation, the disclosed embodiments require developers to maintain fewer mapping functions.

[0143] In an optional embodiment, when converting the second computation graph into a computation graph under a specific deep learning framework, operators whose operator identifiers have changed are identified and marked so that users can understand which operators have been modified after the conversion (including quantized and merged operators), which helps users to identify and distinguish them.

[0144] It should be pointed out that, although not explicitly stated above, those skilled in the art will understand that the computation graph and intermediate representation described above are both static computation graphs.

[0145] The following examples and combined Figure 7The embodiments of the present disclosure are described in detail. The first computation graph 701 includes operators A, B, C, D, E, and F. After operator conversion, it becomes an intermediate expression 702: A', B', C', D', E', and F'. The properties of A', B', C', D', E', and F' are all changed. The computation graph including A', B', C', D', E', and F' is provided to the compiler, which will continue to perform various model processing on it, and ultimately obtain a second computation graph 703 including A', H', G' (B'C'D'), E', and F'. Among them, H' is a newly added operator, which can be a quantization or dequantization operator. G' is a merge operator of B', C', and D' and has the functions of B', C', and D'. To return the second computation graph 703 to the original framework, A', E', and F' need to be mapped back to A, E, and F, and G' is mapped to G", and the resulting computation graph is A, H', G", E, and F. Here, when the operators of the first computation graph are converted to operators of the intermediate representation, generally, the operator identifiers of the operators do not change, but the attributes of the operators change. Therefore, it can be considered that the operator identifiers of A', B', C', D', E', F' and A, B, C, D, E, F in this example are the same.

[0146] Figure 9 A flowchart of a method for processing a computational graph of a deep learning model according to another embodiment of the present disclosure is shown, including steps S901 to S905.

[0147] Step S901 is used to preprocess the computation graph of the deep learning model of a specific deep learning framework. Preprocessing may include sorting the computation graph and removing duplicate operators to ensure unique identifiers of operators in the computation graph.

[0148] Step S902 is used to initialize the data structure. It specifically includes steps S9021 and S9022. S9021 is used to construct an operator mapping table. The operator mapping table can be found in Table 1. Step S9022 is used to construct an attribute storage unit for storing attributes.

[0149] The attribute storage unit for storing data can be implemented in a variety of ways. For example, the storage unit can be a data structure defined in a program, or a variable-length string storage space, and so on.

[0150] Step S903 extracts attributes from the preprocessed computation graph and stores them in an attribute storage unit.

[0151] In this step, the attributes obtained from the preprocessing calculation graph are stored in the attribute storage unit according to the operator identifier, attribute identifier and attribute value. If the attribute storage unit is a variable-length string storage space, the string can be organized into bytes according to the actual storage requirements of the operator identifier, attribute identifier and attribute value.

[0152] Step S904 is operator mapping, that is, S9041 converts the attributes of the operator into the intermediate expression to define the attributes of the corresponding operator.

[0153] To implement this step, you need to define several mapping functions to represent the mapping relationship between the operators in the original computation graph that require attribute conversion and the corresponding operators in the intermediate representation. The mapping functions are executed to convert the original operators in the computation graph into the operators defined in the intermediate representation. The original operators in the computation graph and the corresponding operators defined in the intermediate representation must have the same operator identifier, or the operator identifier must conform to a certain set specification.

[0154] Step S905 is model processing, that is, quantizing, graph slicing, model pruning, compiling, etc. the intermediate expression to obtain a processed intermediate expression.

[0155] Step S906 is attribute restoration, that is, when S9061 converts the intermediate expression back to the original deep learning framework, the current attributes of the corresponding operator are restored to the original attributes using the previously stored attribute data.

[0156] The processed computational graph needs to restore the unique properties of the original deep learning framework. Therefore, this step uses the previously stored attribute data to replace the attribute data of some operators. Here, the attribute data includes attribute identifiers and attribute values.

[0157] In an optional embodiment, step S903 may use a string to store attributes, which will be described below in conjunction with Table 2.

[0158] Table 2

[0159]

[0160] Table 2 shows the attribute data for operators op1 through opn. As shown in Table 2, the order is: Op1 name (op1 name), seperator1 (separator 1), attribute1 (attribute 1), seperator1 (separator 1), value1 (attribute 1's attribute value), seperator2 (separator 2), Op2 name (op2 name), seperator1 (separator 1), attribute1 (attribute 1), seperator1 (separator 1), value1 (attribute 1's attribute value), attribute2 (attribute 2), seperator1 (separator 1), value2 (attribute 2's attribute value), and so on. In Table 2, within the same operator, the name and attributes, as well as the attributes themselves, are separated by seperator 1, while different operators are separated by seperator 2. This allows the attribute data of each operator to be extracted using the seperators.

[0161] Using string storage, storage space can be allocated based on the actual number of characters in the operator identifier, attribute identifier, attribute value, and separator, ensuring that the attributes are stored in the minimum storage space. Of course, other methods can also be used to ensure that the attributes are stored in the minimum storage space, such as using appropriate data structures to achieve this goal.

[0162] When a string storage attribute is used, the string can be stored in the intermediate representation 802. The intermediate representation 802 is provided to, for example, a compiler. The compiler does not process the string when processing the model. After the compiler completes the model processing, the string is still retained in the output computation graph.

[0163] Then in step S905, for the received computation graph, the string is converted into Figure 10 The multi-layer variable container shown in the figure can then be used to quickly locate and restore attributes using the search function provided by the multi-layer variable container. The multi-layer variable container can hierarchically store operator identifiers, attribute identifiers, and attribute values. The first-level container contains the operator identifiers of the operators. The second-level container extends from the first-level container to contain the attribute identifiers of the operators. The third-level container extends from the second-level container to contain the attribute values of the attribute identifiers of the operators. Figure 10 As shown, the first layer of the exemplary multi-layer variable container stores operator identifiers 1 to n, the second layer stores attribute identifiers 1-n, and the third layer stores attribute values 1-n.

[0164] The computation graph processing method provided in this embodiment uses the minimum storage space to store attributes to reduce data storage. During conversion, the original attributes are first converted into a multi-layer variable container, so that the attributes can be restored using the multi-layer variable container to improve recovery efficiency.

[0165] Furthermore, although the above description uses a data center server as an example of an execution subject of the embodiments of the present disclosure, the present disclosure is not limited thereto. Theoretically, the execution subject of the embodiments of the present disclosure can be any computing device, including the servers and terminal devices described above. For terminal devices, as long as the processor, memory, and network throughput capabilities of the terminal device can meet the operating requirements of the deep learning model, the deep learning model can be deployed on it and various computational graph processing (including the computational graph processing solutions provided by the embodiments of the present disclosure) can be performed on it.

[0166] The commercial value of the disclosed embodiments

[0167] Deep learning models currently have a wide range of successful applications, making even small improvements crucial not only from a technical perspective but also from a commercial perspective. For example, in the field of facial recognition, facial images can be identified using deep learning models from video surveillance cameras. By comparing these images with faces stored in the cloud, criminals can be identified in the surveillance footage. Similarly, in the field of speech recognition, deep learning models can be used to perform speech recognition and enable simultaneous interpretation. These applications can generate significant commercial benefits.

[0168] In the engineering practice of deep learning models, it is necessary to convert computational graphs under various frameworks into computational graphs adapted for a specific acceleration unit, and then return the computational graph adapted for the specified acceleration unit to the original framework, thereby organically combining and mutually promoting the model's algorithmic research and engineering application. The computational graph processing method provided in the disclosed embodiment is used to solve the problem of maintaining numerous mapping functions during computational graph conversion, and therefore the disclosed embodiment also has application prospects and commercial value.

[0169] Those skilled in the art will appreciate that the present disclosure can be implemented as systems, methods, and computer program products. Therefore, the present disclosure can be specifically implemented in the following forms, namely, complete hardware, complete software (including firmware, resident software, microcode), and can also be implemented in the form of a combination of software and hardware. In addition, in some embodiments, the present disclosure can also be implemented in the form of a computer program product in one or more computer-readable media, wherein the computer-readable medium contains computer-readable program code.

[0170] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media are, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by a processing unit, device or device, or used in combination with it.

[0171] A computer-readable signal medium may include a data signal propagated in baseband or as part of a chopped wave, which carries computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction system, apparatus, or device.

[0172] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0173] The computer program code for executing the disclosed embodiments can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages, such as JAVA, C++, and can also include conventional procedural programming languages, such as C. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0174] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A processing unit for performing data conversion between a computation graph in a first deep learning framework and an intermediate representation deployed in an acceleration unit having an instruction set architecture, the processing unit comprising: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit, configured to execute the decoded computer instructions to achieve: Extracting properties of operators from the computational graph of a deep learning model under the first deep learning framework; Converting the computation graph into an intermediate expression conforming to the acceleration unit, wherein the attributes of operators in the intermediate expression are different from the attributes of operators in the computation graph; Performing model processing on the intermediate expression; wherein the intermediate expression after model processing is a computational graph deployed to a designated acceleration unit through a running stack; Converting the processed intermediate representation back to a computational graph of the first deep learning framework, wherein, for operators whose operator identifiers have not changed, replacing the attributes of the operator in the intermediate representation with the extracted attributes of the operator; The step of extracting the attributes of the operator from the computation graph of the deep learning model under the first deep learning framework includes: Extracting an operator identifier, an attribute identifier, and an attribute value of an operator from a computational graph of a deep learning model under the first deep learning framework; The extracted attribute identifiers and attribute values are stored in correspondence with the operator identifiers; Storing the extracted attribute identifier and attribute value in correspondence with the operator identifier includes: storing the operator identifier of the operator, the attribute identifier in the operator, and the attribute value thereof as a string; wherein the instruction execution unit further implements: storing the string as a multi-level variable container, wherein a first-level container in the multi-level variable container accommodates the operator identifier of the operator; a second-level container is extended from the first-level container to accommodate the attribute identifier of the operator; and a third-level container is extended from the second-level container to accommodate the attribute value of the attribute identifier of the operator.

2. The processing unit according to claim 1, wherein: The model processing includes at least one of the following: operator merging, model pruning, quantization, and graph slicing.

3. The processing unit according to claim 1, wherein: The method of replacing the attributes of the operator in the intermediate expression with the extracted attributes of the operator includes: according to the operator identifier of the operator, obtaining the attribute identifier and attribute value of the operator in the character string where the operator identifier of the operator is located, and replacing the attribute identifier and attribute value in the operator in the intermediate expression.

4. The processing unit according to claim 3, wherein: There is a first separator between character strings of different operators, and a second separator is respectively provided between the operator identifier and the attribute identifier, and between the attribute identifier and the attribute value in the character string of the same operator.

5. The processing unit according to claim 3, wherein replacing the attribute of the operator in the intermediate expression with the extracted attribute of the operator comprises: According to the operator identifier of the operator, the attribute identifier and attribute value of the operator contained in the second-level container and the third-level container respectively extended from the first-level container where the operator identifier of the operator is located are obtained, and the attribute identifier and attribute value of the operator in the intermediate expression are replaced. The processing unit according to claim 1 , wherein: The instruction execution unit further implements: before the step of extracting the attributes of the operator from the calculation graph of the deep learning model under the first deep learning framework, sorting the calculation graph and removing duplicate operators.

7. The processing unit according to claim 1, wherein: The converting of the computation graph into an intermediate expression conforming to the acceleration unit includes: converting the attributes of at least one operator of the computation graph into attributes of a corresponding operator defined by the intermediate expression through a mapping function.

8. The processing unit according to claim 7, wherein: An operator mapping table is used to organize the mapping function.

9. The processing unit according to claim 1, wherein: The properties of the extracted operators are stored in intermediate representations before and after processing.

10. The processing unit according to claim 1, wherein: The instruction execution unit further implements: when converting the processed intermediate expression back into the computational graph of the first deep learning framework, identifying and marking operators whose operator identifiers have changed.

11. A computing device comprising a memory and the processing unit according to any one of claims 1 to 10.

12. A method for processing a computational graph of a deep learning model, for performing data conversion between a computational graph in a first deep learning framework and an intermediate representation deployed in an acceleration unit having an instruction set architecture, the method comprising: Extracting properties of operators from the computational graph of a deep learning model under the first deep learning framework; Converting the computation graph into an intermediate expression conforming to the acceleration unit, wherein the attributes of operators in the intermediate expression are different from the attributes of operators in the computation graph; Performing model processing on the intermediate expression; wherein the intermediate expression after model processing is a computational graph deployed to a designated acceleration unit through a running stack; Converting the processed intermediate representation back to a computational graph of the first deep learning framework, wherein, for operators whose operator identifiers have not changed, replacing the attributes of the operator in the intermediate representation with the extracted attributes of the operator; The step of extracting the attributes of the operator from the computation graph of the deep learning model under the first deep learning framework includes: Extracting an operator identifier, an attribute identifier, and an attribute value of an operator from a computational graph of a deep learning model under the first deep learning framework; The extracted attribute identifiers and attribute values are stored in correspondence with the operator identifiers; Storing the extracted attribute identifiers and attribute values in correspondence with the operator identifier includes: storing the operator identifier of the operator, the attribute identifier in the operator, and the attribute value thereof as a character string; wherein the character string is stored as a multi-level variable container, wherein a first-level container in the multi-level variable container accommodates the operator identifier of the operator; a second-level container is extended from the first-level container to accommodate the attribute identifier of the operator; and a third-level container is extended from the second-level container to accommodate the attribute value of the attribute identifier of the operator.

13. The computational graph processing method according to claim 12, wherein: The model processing includes at least one of the following: operator merging, model pruning, quantization, and graph slicing.

14. The computational graph processing method according to claim 12, wherein: The replacing the attributes of the operator in the intermediate expression with the extracted attributes of the operator includes: according to the operator identifier of the operator, obtaining the attribute identifier and attribute value of the operator in the character string where the operator identifier of the operator is located, and replacing the attribute identifier and attribute value of the operator in the intermediate expression.

15. The computational graph processing method according to claim 14, wherein: There is a first separator between character strings of different operators, and a second separator is respectively provided between the operator identifier and the attribute identifier, and between the attribute identifier and the attribute value in the character string of the same operator.

16. The computational graph processing method according to claim 12, further comprising: Before the step of extracting the attributes of the operators from the computation graph of the deep learning model under the first deep learning framework, the computation graph is sorted and duplicate operators are removed.

17. The computational graph processing method according to claim 12, wherein: The converting of the computation graph into an intermediate expression conforming to the acceleration unit includes: converting the attributes of at least one operator of the computation graph into attributes of a corresponding operator defined by the intermediate expression through a mapping function.

18. The computational graph processing method according to claim 12, wherein: The properties of the extracted operators are stored in intermediate representations before and after processing.

19. A data center comprising the computing device according to claim 11.

Images