Processing unit, computing device, and computational graph processing method for deep learning model

Abstract

The present invention discloses a method for processing a computational graph of a processing unit, a computing device, and a deep learning model. The method includes: converting a computational graph of a first deep learning framework into an intermediate expression that complies with the acceleration unit; performing model processing on the intermediate expression; determining a first operator contained in the processed intermediate expression and that has not been registered in the first deep learning framework, wherein the processed intermediate expression represents the operator with an operator identifier, attribute data, and a connection relationship with other operators; converting the processed intermediate expression back into the computational graph of the first deep learning framework, including: replacing the first operator with a connection operator, and constructing a connection relationship between the connection operator and other operators according to the connection relationship between the first operator and other operators, and then replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator. The embodiment of the present disclosure uses a connection operator to solve the conversion problem of multiple operators in the computational graph that are not defined and registered in the original framework.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more specifically, to a processing unit, a computing device, and a computational graph processing method for a deep learning model. Background Art

[0002] Currently, mainstream deep learning frameworks include TensorFlow, MxNet, Caffe, and MxNet. These frameworks define different sets of operators, and even operators with similar functionality may have different properties across different frameworks. Operators are the fundamental units of computation in deep learning models, such as the convolution operator. In TensorFlow, the convolution operator's padding attribute is either "VALID" or "SAME," indicating that after the convolution operation, only valid convolution is performed, ignoring data at the edges, while "SAME" preserves the convolution results at the edges. However, in MxNet and Caffe, the convolution operator's padding attribute is a two-dimensional array, allowing users to specify horizontal and vertical padding values. Therefore, two operators with the same identifier may have different properties, making them incompatible. Operators are defined by both identifiers and properties.

[0003] Those skilled in the art are aware that for a deep learning model to run on an acceleration unit, the corresponding processing unit must perform some model processing, which is then compiled into an acceleration unit model supported by the acceleration unit's instruction set. This model processing includes operator merging and quantization. Operator merging is the process of fusing operators within a deep learning model, for example, combining multiple operators into a single operator. Quantization is the process of converting parameters such as weights in a deep learning model, as well as inputs to the deep learning model, from high-precision data types to low-precision data types. These model processing operations must be performed within a framework or instruction set supported by the acceleration unit. These only support the attributes and attribute values ​​of each operator within the acceleration unit's framework, not the attributes and attribute values ​​of each operator within the original deep learning framework of the deep learning model. Therefore, the deep learning model within the original deep learning framework must be converted into an intermediate representation supported by the acceleration unit before model processing within the framework supported by the acceleration unit can be performed. The intermediate representation after model processing is then converted back into the computational graph of the original framework.

[0004] Converting the intermediate representation after model processing back to the computational graph of the original framework involves converting the operators represented in the intermediate representation into operators that can be recognized by the original framework. Since operators are represented in the computational graph using operator identifiers, attribute data, and relationships with other operators, the operator identifiers, attribute data, and relationships with other operators of each operator after conversion must conform to the definitions of the original framework. To this end, operators included in the intermediate representation but not defined in the original framework must be defined and registered in the original framework. Defining each such operator separately in the original framework requires developers to develop and maintain multiple new operators in the original framework, which increases maintenance costs. Summary of the Invention

[0005] Based on this, the purpose of the present disclosure is to provide a processing unit, a computing device and a computational graph processing method for a deep learning model to solve the problems existing in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a processing unit, including:

[0007] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0008] an instruction decoding unit for decoding the retrieved computer instructions;

[0009] An instruction execution unit, configured to execute the decoded computer instructions to achieve:

[0010] Converting a computational graph of a deep learning model used in a first deep learning framework into an intermediate expression conforming to the acceleration unit;

[0011] performing model processing on the intermediate expression;

[0012] Determining a first operator included in the processed intermediate expression and not registered in the first deep learning framework, wherein the processed intermediate expression represents the operator using an operator identifier, attribute data, and a connection relationship with other operators;

[0013] Converting the processed intermediate expression back into a computational graph of the first deep learning framework includes:

[0014] In the processed intermediate expression, the first operator is replaced by a connection operator, and the connection relationship between the connection operator and other operators is constructed according to the connection relationship between the first operator and other operators, and then the operator identifier and attribute data of the connection operator are replaced by the operator identifier and attribute data of the first operator.

[0015] Optionally, the operator identifier and attribute data of the first operator are stored in the attribute data of the join operator, and replacing the operator identifier and attribute data of the join operator with the operator identifier and attribute data of the first operator includes:

[0016] The operator identifier and attribute data of the first operator are read from the attribute data of the connection operator, and the operator identifier and attribute data of the connection operator are replaced respectively.

[0017] Optionally, the step of replacing the first operator with the connection operator is repeated until all the first operators in the processed intermediate expression are replaced; then the step of replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator is repeated until all the connection operators in the processed intermediate expression are replaced.

[0018] Optionally, the model processing includes at least one of the following: operator merging, quantization, graph slicing, and model pruning.

[0019] Optionally, the instruction execution unit further implements:

[0020] Before replacing the first operator with the join operator, converting the processed intermediate expression into a JSON file; and

[0021] After replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator, converting the json file into a format of a computational graph followed by the first deep learning framework.

[0022] Optionally, the first deep learning framework is an MxNet framework, and the connection operator is a registration operator of the MxNet framework.

[0023] Optionally, the constructor of the connection operator specifies other operators that have a connection relationship with the first operator through input and / or output parameters, so that the constructor of the connection operator can construct the connection relationship between the connection operator and other operators according to the connection relationship between the first operator and other operators.

[0024] Optionally, the constructor of the connection operator further stores the number of input tensors and the number of output tensors of the first operator in the attribute data of the connection operator.

[0025] Optionally, converting the computation graph into an intermediate expression conforming to the acceleration unit includes: converting attribute data of at least one operator of the computation graph into attribute data of a corresponding operator defined by the intermediate expression through a mapping function.

[0026] Optionally, determining that the processed intermediate expression contains a first operator that has not been registered in the first deep learning framework includes: comparing the operator identifier of each operator in the processed intermediate expression with the operator identifier of the registered operator under the first deep learning framework to determine the first operator.

[0027] In a second aspect, an embodiment of the present disclosure provides a computing device comprising a memory and any one of the processing units described above.

[0028] Optionally, the computing device is a server or a terminal device.

[0029] In a third aspect, an embodiment of the present disclosure provides a method for processing a computational graph of a deep learning model, including:

[0030] Converting a computational graph of a deep learning model used in a first deep learning framework into an intermediate expression conforming to the acceleration unit;

[0031] performing model processing on the intermediate expression;

[0032] Determining a first operator included in the processed intermediate expression and not registered in the first deep learning framework, wherein the processed intermediate expression represents the operator using an operator identifier, attribute data, and a connection relationship with other operators;

[0033] Converting the processed intermediate expression back into a computational graph of the first deep learning framework includes:

[0034] In the processed intermediate expression, the first operator is replaced by a connection operator, and the connection relationship between the connection operator and other operators is constructed according to the connection relationship between the first operator and other operators, and then the operator identifier and attribute data of the connection operator are replaced by the operator identifier and attribute data of the first operator.

[0035] Optionally, the operator identifier and attribute data of the first operator are stored in the attribute data of the join operator, and replacing the operator identifier and attribute data of the join operator with the operator identifier and attribute data of the first operator includes:

[0036] The operator identifier and attribute data of the first operator are read from the attribute data of the connection operator, and the operator identifier and attribute data of the connection operator are replaced respectively.

[0037] Optionally, the step of replacing the first operator with the connection operator is repeated until all the first operators in the processed intermediate expression are replaced; then the step of replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator is repeated until all the connection operators in the processed intermediate expression are replaced.

[0038] Optionally, the model processing includes at least one of the following: operator merging, quantization, graph slicing, and model pruning.

[0039] Optionally, it also includes:

[0040] Before replacing the first operator with the join operator, converting the processed intermediate expression into a JSON file; and

[0041] After replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator, converting the json file into a format of a computational graph followed by the first deep learning framework.

[0042] Optionally, the first deep learning framework is an MxNet framework, and the connection operator is a registration operator of the MxNet framework.

[0043] Optionally, the constructor of the connection operator specifies other operators that have a connection relationship with the first operator through input and / or output parameters, so that the constructor of the connection operator can construct the connection relationship between the connection operator and other operators according to the connection relationship between the first operator and other operators.

[0044] Optionally, the constructor of the connection operator further stores the number of input tensors and the number of output tensors of the first operator in the attribute data of the connection operator.

[0045] In a fourth aspect, an embodiment of the present disclosure provides a data center, comprising the above-mentioned computing device serving as a server.

[0046] Different from the prior art, in which each operator contained in the intermediate expression but not defined in the original framework needs to be defined separately in the original framework, the embodiment of the present disclosure uses only one operator, the connection operator, to replace all these operators contained in the intermediate expression but not defined and registered in the original framework, thereby greatly reducing the maintenance cost of the new operator. The connection operator has the same connection relationship with other operators as the operators contained in the intermediate expression but not defined in the original framework, but the attribute data and operator identifier of the connection operator are different from those of the operators contained in the intermediate expression but not defined in the original framework. The attribute data and operator identifier of the connection operator need to be replaced with the attribute data and operator identifier of the operator contained in the intermediate expression but not defined in the original framework. According to the embodiment of the present disclosure, developers only need to define and register one connection operator to solve the conversion problem of multiple operators in the calculation graph that are not defined and registered in the original framework, thereby avoiding the development and maintenance of multiple new operators in the original framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0048] Figure 1 A hierarchical structure diagram of a data center to which an embodiment of the present disclosure is applied is shown;

[0049] Figure 2 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0050] Figure 3 This is a block diagram of the internal structure of a server in a data center according to an embodiment of the present disclosure;

[0051] Figure 4 is a control relationship diagram of a central processing unit (CPU) and a neural network acceleration unit (acceleration unit) inside a server according to one embodiment of the present disclosure;

[0052] Figure 5 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present disclosure;

[0053] Figure 6 It is a layered software architecture diagram;

[0054] Figure 7 is an example graph of computational graph transformation;

[0055] Figure 8 This is a partial flow chart of a computation graph processing method provided by an embodiment of the present disclosure;

[0056] Figure 9 is a partial flow chart of a computational graph processing method provided by another embodiment of the present disclosure;

[0057] Figure 10 It is a schematic diagram of the operator connection relationship. DETAILED DESCRIPTION

[0058] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0059] The following terms are used in this article.

[0060] Accelerator unit: A processing unit designed to improve data processing speed in specialized areas (e.g., image processing, neural network operations, etc.) where general-purpose processors are inefficient. It is often used in conjunction with a general-purpose processor (CPU), controlled by the general-purpose processor, to perform processing for specific purposes or fields, thereby improving computer processing efficiency in those areas. It can also be called an AI processing unit and can include a graphics processing unit (GPU), a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated AI acceleration hardware (e.g., an accelerator unit).

[0061] On-chip memory: Memory that is used exclusively within the main core or sub-core and cannot be shared.

[0062] The command processor serves as the command interface between the accelerator unit and the central processing unit (CPU), which drives it. The command processor receives instructions from the CPU for the accelerator unit and distributes them to the various cores within the accelerator unit for execution. It also manages the synchronization of the cores within the accelerator unit.

[0063] Lifecycle: An operand isn't always involved in the entire instruction sequence. The period between its first appearance and its last use in the instruction sequence is the lifecycle of the operand. In other words, after its lifecycle, it's no longer needed and doesn't need to be stored in on-chip memory.

[0064] Neural network: Generally refers to an artificial neural network (ANN), an algorithmic network that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. A classic neural network, also the simplest neural network structure, consists of three layers: input layer, output layer, and middle layer (also called hidden layer). The input layer, output layer, and middle layer each contain multiple nodes.

[0065] Neural network model: In a neural network, nodes are mathematized to generate mathematical models of nodes. The mathematical models of a large number of nodes in a neural network constitute the neural network model.

[0066] Deep learning model: The concept of deep learning originates from the study of neural networks. A neural network with multiple intermediate layers is called a deep learning network. Therefore, in this sense, a deep learning model is also a type of neural network model. Both deep learning models and neural network models must be trained. Sample data is input into a designed network structure (i.e., the network structure has been determined), feature information is extracted through multiple intermediate layers, and the weight parameters of the neurons are continuously modified based on the output results of the output layer, so that the output results of the output layer increasingly converge to the preset results until the final weight parameters are determined. The trained deep learning model can be truly applied in real-world scenarios. At the same time, the use of the deep learning model in real-world scenarios can also be collected to optimize the deep learning model.

[0067] Node: The smallest unit of independent operation in a deep learning model, which receives input and generates output after operating on its own weight parameters or parameters in other models (such as hyperparameters). A deep learning model can include various specific operations such as convolution and pooling, and has various operation nodes including convolution nodes and pooling nodes. There are multiple layers in a deep learning model, each layer has multiple nodes, and the output of each node is the input of the node in the next layer. Specifically, a node includes a program for a specific operation and related data. For example, a convolution operation node includes the program code used for the convolution operation and some data used in the convolution.

[0068] An operator is a collection of operations built into a deep learning model to implement specific functionality. Each layer of a deep learning model can contain multiple such operators. These are referred to as operations in the TensorFlow framework and layers in the Caffe framework. Operators are considered a further abstraction based on nodes; an operator can correspond to one or more nodes. Therefore, operators and nodes can sometimes represent the same program code.

[0069] Instruction set: A collection of instructions for operations supported within the chip, such as operations that mainly support deep learning operators, such as Convolution, Pooling, ROI, etc.

[0070] Quantization: The act of converting the inputs of operation nodes in a deep learning model, as well as the weight parameters and other parameters of the operation nodes, from high-precision data types to low-precision data types, thereby reducing the requirements for data throughput and storage space.

[0071] Dequantization: The opposite process of quantization, which is the process of converting the input of the operation node in the deep learning model, as well as the weight parameters and other parameters of the operation node from low-precision data types to high-precision data types.

[0072] Intermediate Representation (IR): Deep learning models have different formats, depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet. The code representations of these deep learning models also vary. This poses significant challenges to the universality of deep learning model quantization. An intermediate representation (IR) converts deep learning model code in various formats into a single representation that conforms to one or more acceleration units. It analyzes the meaning of each code statement in a deep learning model and translates it into a common representation based on its meaning. This ensures that code statements with the same meaning across different deep learning models are expressed identically in the IR. Currently, there are tools available that convert the representations of different deep learning models into IR.

[0073] Computation graph: Currently, deep learning frameworks mainly have two programming methods: declarative programming and imperative programming. In declarative programming, the program code first defines a neural network model structure to describe the calculation logic, but it is not executed immediately. The neural network model structure is only executed when the program code that calls the network model structure is executed. The neural network model structure includes multiple operators (or symbolic expressions of operators) and the connection relationship between them, and can be shown graphically. Therefore, the neural network model structure is called a static computation graph. In imperative programming, the program code directly returns the result of the operation, and the definition and execution of the neural network model structure are synchronized. Generally speaking, static graphs are convenient for optimizing the overall neural network model and are more conducive to performance improvement, while dynamic graphs are very convenient for users to debug specific programs.

[0074] Figure 1 A hierarchical structure diagram of a data center is shown as a scenario in which an embodiment of the present disclosure is applied.

[0075] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Neural networks, a key AI technology, are already widely used in data center big data analytics.

[0076] In traditional large data centers, the network structure is usually Figure 1 The three-layer structure shown is a hierarchical inter-networking model. This model consists of the following three layers:

[0077] Access Layer 103: Sometimes also called the edge layer, it includes access switches 130 and the servers 140 connected to them. Servers 140 are the processing and storage entities of the data center, processing and storing large amounts of data within the data center. Access switches 130 connect these servers to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, hence the name "top of rack" switches, physically connecting the servers.

[0078] Aggregation Layer 102: sometimes also called the distribution layer, includes aggregation switches 120. Each aggregation switch 120 connects to multiple access switches and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0079] Core Layer 101: Includes core switch 110. Core switch 110 provides high-speed forwarding for packets entering and exiting the data center and provides connectivity for multiple aggregation layers. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 provides a resilient L3 routing network for the entire data center network.

[0080] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0081] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with the remaining aggregation switches 120 being used only in the event of a failure (as indicated by the dotted lines in the diagram above). This means that horizontal scalability is impossible at the aggregation layer because, even if multiple aggregation switches 120 are added, only one is still operational.

[0082] Figure 2 Shown Figure 1 The physical connection of each component in the hierarchical data center. Figure 2 As shown, one core switch 110 is connected to multiple aggregation switches 120 , one aggregation switch 120 is connected to multiple access switches 130 , and one access switch 130 is connected to multiple servers 140 .

[0083] server

[0084] Since the server 140 is the real execution entity of the data center, Figure 3 The following figure shows a block diagram of the internal structure of a server 140. Server 140 includes a bus-connected memory 210, a central processing unit (CPU) 220, and various acceleration units. These acceleration units include an embedded neural network processor (acceleration unit) 230, a data transfer unit (DTU) 260, a graphics processing unit (GPU, not shown), an application-specific integrated circuit (ASIC, not shown), and a field-programmable gate array (FPGA, not shown).

[0085] The traditional processor architecture design results in control units and storage units occupying a large portion of the space, while the computing unit occupies less space. As a result, while highly effective for logic control, it is inefficient for large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively increase the speed of computation for different functions and fields. The acceleration unit proposed in this disclosure can be any of these types, and each of these acceleration units is described below.

[0086] Accelerator unit 230: This is a processing unit that utilizes a data-driven parallel computing architecture to handle the large number of operations (e.g., convolution, pooling, etc.) within each neural network node. Because the data and intermediate results from these operations (e.g., convolution, pooling, etc.) are closely linked and frequently used throughout the computational process, existing CPU architectures require frequent access to external memory due to the limited internal memory capacity of the CPU core, resulting in inefficient processing. By implementing an acceleration unit, each core has on-chip memory suitable for the storage capacity required for neural network computations, eliminating the need for frequent access to external memory, significantly improving processing efficiency and computational performance.

[0087] Data Transfer Unit (DTU) 260: This is a wireless terminal device specifically designed to convert serial port data into IP data or vice versa for transmission over a wireless communication network. The DTU's primary function is to wirelessly transmit data from remote devices back to the backend center. At the front end, the DTU and the client's device are connected via an interface. After powering on, the DTU first registers with the mobile GPRS network and then establishes a socket connection with the backend center installed in the DTU. The backend center acts as the server of the socket connection, and the DTU is the client of the socket connection. Therefore, the DTU is used in conjunction with the backend software. Once the connection is established, the front-end device and the backend center can perform wireless data transmission via the DTU.

[0088] A graphics processing unit (GPU) is a microprocessor specifically designed for image and graphics-related computing. The GPU addresses the CPU's lack of computing space by utilizing a large number of dedicated graphics processing units. This reduces the graphics card's reliance on the CPU and allows it to handle some of the computationally intensive image processing tasks previously handled by the CPU.

[0089] Application-Specific Integrated Circuit (ASIC): An integrated circuit designed and manufactured to meet specific user requirements and the needs of a specific electronic system. Because this type of integrated circuit is customized according to user requirements, its structure is often adapted to that specific user's needs.

[0090] Field Programmable Gate Array (FPGA): A further development of programmable devices such as PALs and GALs. It emerged as a semi-custom circuit within the field of Application-Specific Integrated Circuits (ASICs), addressing both the shortcomings of custom circuits and the limited number of gates inherent in existing programmable devices.

[0091] Although the acceleration unit has the advantage of being much more efficient than an ordinary processor for specific applications or fields, it is also subject to the control of the processing unit 220. Taking the acceleration unit dedicated to deep learning models as an example, the memory 210 stores various deep learning models, including the neurons and neuron weight data of these models. These deep learning models are called when needed. Figure 3 A processing unit 220 is deployed to an acceleration unit 230. Specifically, the processing unit 220 can inform the acceleration unit 230 of the storage location of the deep learning model in memory 210 in the form of instructions. The acceleration unit 230 can then address these locations and store the instructions to be executed in its on-chip memory. The processing unit 220 can also send instructions to be executed by the acceleration unit 230 in the form of instructions. The acceleration unit 230 receives the instructions and stores them in the on-chip memory. Similarly, the acceleration unit 230 can also obtain input data in the above manner. The acceleration unit 230 obtains the instructions to be executed and the input data to perform inference calculations. The node weight parameters can be included in the instruction sequence of the deep learning model and retrieved from memory 210 by the acceleration unit 230. Of course, the node weight parameters can also be stored independently and retrieved from memory 210 by the acceleration unit 230 when needed. Here, the processing unit 220 can be understood as a hardware unit with scheduling and control capabilities, generally a hardware unit such as a central processing unit (CPU), microcontroller, or microprocessor.

[0092] Internal structure of the processing unit and acceleration unit 230

[0093] The following combination Figure 4 The internal structure diagram of the processing unit and the acceleration unit 230 is shown, which illustrates how the processing unit controls the operation of the acceleration unit.

[0094] like Figure 4 As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0095] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 4 A register in the register file 229 shown is used to store instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, incrementing the address or decrementing the address according to the instruction length.

[0096] After fetching the instruction, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0097] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0098] After the instruction issue unit 225 sends the instruction to the instruction execution unit 226, the instruction execution unit 226 begins executing the instruction. However, if the instruction execution unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is for neural network inference, the instruction execution unit 226 will not execute the instruction but instead send it via the bus to the acceleration unit 230 for execution.

[0099] The acceleration unit 230 includes multiple cores 236 ( Figure 4 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0100] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus. According to different mechanisms, the bus channel 231 may include a PCIE channel 232, an I2C channel 233, and a JTAG channel 234.

[0101] PCIE, or PCI-Express, is a high-speed serial computer expansion bus standard proposed by Intel in 2001 to replace the old PCI, PCI-X, and AGP bus standards. PCIE is a high-speed serial point-to-point dual-channel high-bandwidth transmission. The connected devices are allocated exclusive channel bandwidth and do not share bus bandwidth. It mainly supports active power management, error reporting, end-to-end reliable transmission, hot plugging, and quality of service functions. Its main advantage is its high data transmission rate, and it also has considerable development potential. At present, most PCIE buses are PCIE GEN3, but the disclosed embodiments can also use PCIE GEN4, that is, a bus channel that complies with the PCI-Express 4.0 standard.

[0102] I2C channel 233 is a simple, bidirectional, two-wire synchronous serial bus developed by Philips. It only requires two wires to transmit information between devices connected to the bus.

[0103] JTAG stands for Joint Test Action Group and is the common name for IEEE Standard 1149.1, Standard Test Access Port and Boundary-Scan Architecture. This standard is used to verify the functionality of designed and manufactured printed circuit boards. JTAG was formally standardized in 1990 by IEEE Document 1149.1-1990. In 1994, a supplementary document describing the Boundary-Scan Description Language (BSDL) was added. Since then, this standard has been widely adopted by electronics companies worldwide. Boundary scan has become almost synonymous with JTAG. JTAG Channel 234 is a bus channel that complies with this standard.

[0104] The direct memory access (DMA) mechanism 235 is a function provided by some computer bus architectures that enables data to be written directly from an attached device (such as external memory) to the on-chip memory of the acceleration unit 230. Compared to the method in which all data transfers between devices must pass through the command processor 237, this method greatly improves the efficiency of data access. It is precisely because of this mechanism that the cores of the acceleration unit 230 can directly access the memory 210 and read parameters in the deep learning model (such as the weight parameters of each node), greatly improving data access efficiency. Although the direct memory access mechanism 235 is shown in the figure as being located between the processor 237 and the bus channel 231, the design of the acceleration unit 230 is not limited to this. In some hardware designs, each core 236 can include a direct memory access mechanism 235, so that the core 236 does not need to go through the command processor 237, but can directly read data from the attached device and write it to the on-chip memory of the acceleration unit 230.

[0105] The command processor 237 assigns instructions sent from the processing unit 220 to the acceleration unit 230 to the core 236 for execution. The instruction execution unit 226 sends pending instructions that require execution by the acceleration unit 230 to the acceleration unit 230, or the instruction execution unit 226 informs the acceleration unit 230 of the storage location of the pending instructions in the memory 210. After the pending instruction sequence enters the bus channel 231, it is cached in the command processor 237, which selects a core 236 and assigns the instruction sequence to it for execution. The pending instructions come from the compiled deep learning model. It should be understood that the pending instruction sequence can include pending instructions executed by the processing unit 220 and pending instructions that need to be executed by the acceleration unit 230.

[0106] Acceleration unit core

[0107] Figure 5 FIG. 4 is a diagram of the internal structure of an acceleration unit core according to an embodiment of the present disclosure.

[0108] In one embodiment, Figure 5 As shown, core 236 includes a tensor engine 310, a pooling engine 320, a convolution process 330, an activation operation 380, a sequencer 350, an instruction cache 340, an on-chip memory 360, and a constant buffer 370. Among them, tensor engine 310, pooling engine 320, convolution process 330, and activation operation 380 are all hardware execution units. Hardware execution units are the hardware modules that actually perform various operations. Some other hardware execution units are not shown in the figure.

[0109] The instruction sequence assigned by the command processor 237 to the core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and distributes them to the various hardware execution units for execution based on the nature of the instructions. The tensor engine 310 is responsible for processing tensor-related operations in deep learning models. The pooling engine 320 is responsible for processing pooling operations in deep learning models. The convolution processing 330 is responsible for convolution operations in deep learning models. The activation operation 380 is used to perform operations corresponding to activation functions in deep learning models. The sequencer 350 determines the distribution of the instructions to the various hardware execution units based on the nature of the fetched instructions, such as convolution, matrix multiplication, or pooling.

[0110] The on-chip memory 360 is an in-core memory that stores the weight parameters in the deep learning model, as well as the inputs and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning model (for example, hyperparameters in the deep learning model) in addition to the weight parameters. As described above, when the processing unit 220 pre-configures the deep learning model in the acceleration unit 230, the processing unit 220 can send the location of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the acceleration unit 230 will retrieve them from the corresponding location in the memory 210 when needed and place them in the on-chip memory 360. For other parameters, the acceleration unit 230 will retrieve them from the corresponding location in the memory 210 when needed and place them in the constant buffer 370. In addition, when the executable instruction is assigned to the core 236 for execution by the command processor 237, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0111] Software architecture diagram

[0112] Improvements to deep learning models require not only the aforementioned hardware support but also continuous improvements at the software and algorithm levels. Only by combining underlying hardware support with the deep learning algorithm architecture above can a powerful computing engine be delivered.

[0113] Figure 6 This is a diagram of a layered software architecture. Layered software design is the mainstream design approach for large-scale software projects. It reduces dependencies between layers, allowing developers to focus on a single layer within the overall structure and easily replace existing layer implementations with new code.

[0114] As shown in the figure, from top to bottom, the software architecture diagram includes an application layer 401, a framework layer 402 and a function layer.

[0115] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, and recommendation 407. These applications are built using this architecture and can also call the architecture to provide runtime interfaces to gain reasoning capabilities within the application.

[0116] The framework layer 402 integrates various deep learning frameworks, such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to enable continued optimization and improvement of various algorithms. TensorFlow 408 is a symbolic mathematics system based on data flow programming and is widely used in programming various machine learning algorithms. MXNet 409 is the deep learning library of choice for Amazon. Caffe 410, or Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modularity.

[0117] The functional layer includes a compilation stack 403 and a runtime stack 404. The compilation stack 403 is used for various model conversions (converter 411), quantization 412, optimization 413, and compilation 414. Conversion 411 converts the model's internal data into an intermediate representation (IR) format. Quantization 412 converts parameters such as weights in a deep learning model, as well as inputs to the deep learning model, from high-precision data types to low-precision data types. Optimization 413 fuses operators within the model and performs operations such as multi-model optimization linking. Compilation 414 optimizes the model based on hardware (e.g., a neural network processor) to generate a binary model that the hardware can recognize. The runtime stack 404 includes a runtime API 415, an execution manager 416, a user-mode driver 417, and a kernel-mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized runtime API 415 provides various runtime callable interfaces. The user-mode driver 417 provides kernel-mode hardware commands and resource scheduling. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0118] By integrating multiple mainstream deep learning models into a single open source platform, developers can develop, compile, and run multiple deep learning models on the same platform, eliminating the need to deploy and maintain multiple model frameworks. Furthermore, the open source platform can be expanded to support more deep learning models.

[0119] Computational Graph Conversion

[0120] Computational graph conversion converts a computational graph from one standard to another. Different standards mean that different deep learning frameworks have different representations of computational graphs. Deep learning frameworks provide integrated environments that support computational graph compilation and execution. Computational graph conversion is required when a computational graph from one framework needs to be processed in another framework.

[0121] In particular, a large number of accelerator units with different instruction set architectures have emerged. To deploy the computation graph of a specific deep learning framework to an accelerator unit with a specific instruction set architecture, the computation graph must be adapted and optimized for that accelerator unit. This is where the intermediate representation comes in. An intermediate representation is a representation of the computation graph defined according to the instruction set architecture of a specific accelerator unit. Computation graphs from different frameworks are first converted to an intermediate representation, and then developers focus on performing model processing on the intermediate representation for the specific accelerator unit. At the same time, the processed intermediate representation sometimes needs to be returned to the original framework for continued training and model improvement. Therefore, the optimized intermediate representation needs to be converted again and converted to the framework's computation graph. This process can be summarized as: converting the computation graph of the original framework to the intermediate representation -> performing model processing on the intermediate representation -> converting the processed intermediate representation to the computation graph of the original framework. This process can be repeated.

[0122] The following describes computational graph conversion based on the aforementioned software architecture diagram. The framework layer 402 can provide computational graphs for deep learning models in various frameworks to the compilation stack 403 or application layer 401. After receiving the computational graph, the compilation stack 403 converts it into an intermediate representation, which is then optimized and quantized. The optimized and quantized intermediate representation can then be deployed to a designated acceleration unit via the runtime stack 404. All computational graphs generated during this process are static.

[0123] Continue to refer Figure 7 As shown. First, as shown in the figure, various deep learning frameworks support models A to M, and the first computation graph 701 of a specific deep learning framework is converted into an intermediate expression 702. The intermediate expression 702 predetermines many operators and their attributes. The operators and attributes of the intermediate expression 702 are defined according to the instruction set of the specified acceleration unit. Converting the first computation graph 701 into the intermediate expression 702 includes: each operator and its attributes in the first computation graph 701 are converted into the corresponding operator and its attributes defined by the intermediate expression 702. This conversion can be achieved with the help of conversion 411. Conversion 411 defines a mapping function to achieve operator conversion. The mapping function is used to convert between operators with the same function but different attributes. Developers understand the specific functions of each operator and their respective attribute definitions, and predefine mapping functions for operators with the same function but different attributes. Referring to the operator mapping table shown in Table 1, the left column is the operator identifier of the first operator, and the right column is the name of the mapping function.

[0124] Table 1

[0125]

[0126] Of course, the embodiments of the present disclosure are not limited to using a mapping function to perform operator attribute conversion, and other methods may also be used to complete operator attribute conversion.

[0127] Then, the intermediate expression 702 is subjected to model processing, including quantization, graph slicing, model pruning, operator merging and other processing. After completion, the intermediate expression is compiled and the second computational graph 703 is output. The quantization processing inserts quantization nodes and dequantization nodes into the intermediate expression 702. Operator merging merges two or more operators into one operator according to the hardware. Graph slicing is to divide the intermediate expression into several subgraphs to facilitate reading and processing by the acceleration unit. Model pruning is a model compression method that introduces sparsity to the dense connections of the deep learning model and reduces the number of non-zero weights by directly setting "unimportant" weights to zero. The second computational graph 703 can be deployed to a designated acceleration unit for execution.

[0128] Finally, if necessary, the second computation graph 703 is converted back to a computation graph for a specific deep learning framework, specifically by converting all operators and their attributes in the second computation graph 703 into operators and their attributes defined by the specific deep learning framework. This conversion can also be achieved through a mapping function. Of course, this step can also be implemented in other ways, for example, by extracting attribute data of the native attributes of each operator from the first computation graph 701. For the second computation graph after model processing, the stored native attributes can be used to replace the current attributes of some operators to achieve a more convenient conversion of operator attributes.

[0129] The following is an example. The first computation graph 701 includes operators A, B, C, D, E, and F. After operator conversion, it becomes an intermediate expression 702: A', B', C', D', E', and F'. Among them, the properties of A', B', C', D', E', and F' are all changed. The computation graph including A', B', C', D', E', and F' is provided to the compiler. The compiler will continue to perform various processing on it, such as quantization, graph slicing, operator merging, model branch reduction, etc., and finally obtains A', H', G' (B' In the second computation graph 703, H' is a newly added operator, which can be a quantization or dequantization operator. G' is a merge operator of B', C', and D', and has the functions of B', C', and D'. To return the second computation graph 703 to the original framework, A', E', and F' need to be mapped back to A, E, and F, and G' needs to be mapped to G", resulting in a computation graph of A, H', G", E, and F. Typically, the operator identifiers of A', B', C', D', E', and F' are the same as those of A, B, C, D, E, and F, or conform to certain set specifications.

[0130] In addition, since the computation graph contains various operators and the connection relationships between them, after the operator conversion, it is also necessary to ensure that the connection relationships between the operators are correct, or the connection relationships between the operators can be directly reconstructed.

[0131] Figure 8 This is a partial flow chart of a computational graph processing method according to an embodiment of the present disclosure. Figure 8 As shown, the computational graph processing method includes the following steps.

[0132] Step S801 reads operators from the computation graph in sequence. The computation graph in this step is the processed general intermediate expression, i.e. Figure 7 The second computation graph 703 in .

[0133] Step S802 determines whether the specified framework has a registered operator with the same operator ID as the current operator. If so, step S804 is executed; if not, step S803 is executed. This step compares the operator ID of the current operator with the operator IDs of all registered operators in the specified framework to determine whether there is a registered operator with the same operator ID in the specified framework. If the operator IDs are the same, regardless of whether the operator attribute data is the same, it is considered that there is a registered operator with the same operator ID in the specified framework.

[0134] Step S803 builds a connection operator and replaces the current operator. The connection operator has the same connection relationship as the current operator. The connection relationship of the operator can be Figure 10 The example shown provides an intuitive understanding. As shown in the figure, the connection relationship of operator D is: three inputs AC and two outputs. When constructing a connection operator, the input and output information is obtained from the current operator, and the connection relationship of the connection operator is constructed based on this input and output information. Furthermore, when constructing the connection operator, the operator identifier and attribute data of the current operator are stored as the attribute data of the connection operator.

[0135] Step S804 calls a mapping function to convert the attribute data of the current operator.

[0136] Step S805 determines whether all operators in the computation graph have been processed. If so, step S806 is executed; otherwise, step S801 is executed.

[0137] According to the above steps, when the specified framework does not support the current operator (the specified framework does not define the current operator), step S803 is called to construct a connection operator to replace the current operator. The connection operator is a registered operator under the specified framework. In a static computation graph, each operator is a character expression. This character expression specifies each operator's operator identifier, attribute data, and connection relationship with other operators. In this article, when referring to an operator in the computation graph, we are actually referring to the character expression that represents the operator in the computation graph. Similarly, the character expression of the connection operator also specifies its own operator identifier, attribute data, and connection relationship. The character expression of the connection operator is generated by the constructor of the connection operator. The constructor of the connection operator specifies other operators that have a connection relationship with the current operator through input and / or output parameters, so that the constructor of the connection operator can construct the connection relationship between the connection operator and other operators according to the connection relationship between the current operator and other operators. At the same time, the constructor of the connection operator also uses the operator identifier and attribute data of the current operator as the attribute data of the connection operator. Finally, the constructor of the connection operator is called to generate the character expression of the connection operator and replace the character expression of the current operator in the computation graph.

[0138] According to the above steps, if the specified framework supports the current operator, as previously described, the current operator and the corresponding registered operator in the specified framework have the same operator identifier, but may have different attribute data. Regardless of whether the attribute data is the same, the operator attributes can be directly converted using a mapping function. The mapping function is used to copy the attribute data of the corresponding registered operator in the specified framework to the attribute data of the current operator. Of course, it is also possible to first determine whether the attribute data of the two is the same, and then use the mapping function to convert the operator attributes if the attribute data is different.

[0139] Based on steps S801-S805, each operator in the computation graph is processed accordingly, resulting in a converted computation graph. This computation graph replaces operators not supported by the specified framework with connection operators and completes the attribute conversion of operators supported by the specified framework. This computation graph is then provided to step S806 for further processing.

[0140] Step S806 reads the connection operators in the calculation graph in sequence.

[0141] Step S807 reads the operator identifier and attribute data of the current operator from the attribute data for the current connection operator, and replaces the operator identifier and attribute data of the connection operator respectively.

[0142] Step S808: Determine whether all connection operators have been processed. If yes, output the computation graph; if not, jump to step S806.

[0143] As mentioned above, each connection operator stores the operator identifier and attribute data of the current operator it replaces in its own attribute data. Therefore, in steps S806-S808, for each connection operator in the calculation graph, the operator identifier and attribute data of the current operator are read from its attribute data, and the operator identifier and attribute data of the connection operator are replaced respectively. After completing the replacement of all connection operators in the calculation graph, the final calculation graph will be able to run and compile under the specified framework.

[0144] This embodiment constructs a connection operator with the same connection relationship as the current operator (there is no registered operator with the same operator identifier under the specified framework) and stores the operator identifier and attribute data of the current operator as the attribute data of the connection operator. The current operator is then replaced with the connection operator. Subsequently, only the operator identifier and attribute data of the connection operator need to be replaced with the operator identifier and attribute data of the current operator. In this way, the corresponding operator can be replaced with a single connection operator. Developers only need to maintain one connection operator, which helps reduce maintenance costs.

[0145] As a variation of the above embodiment, for each current operator not supported by a specified framework, a join operator is constructed to replace the current operator. However, this join operator has the same join relationship as the current operator. However, the operator identifier and attribute data of the current operator are not stored as the attribute data of the join operator. Instead, the corresponding relationship between the operator identifier and attribute data of the current operator and the join operator (for example, the corresponding relationship includes the operator identifier and attribute data of the current operator and the location information of the join operator) is stored elsewhere. Therefore, after each current operator not supported by a specified framework in the computation graph is replaced with a join operator, the operator identifier and attribute data of the current operator corresponding to each join operator are extracted from the corresponding relationship to replace the operator identifier and attribute data of the join operator.

[0146] It should be emphasized that although the processing method of the above embodiment is: first replace the current operators not supported by the specified framework with connection operators one by one until all such operators are replaced, and then replace the operator identifiers and attribute data of the connection operators with the operator identifiers and attribute data of the current operators not supported by the specified framework one by one until all connection operators are replaced. However, the embodiments of the present disclosure are not limited to this. For example, it is also possible to process it like this: first replace the current operators not supported by the specified framework with connection operators, and then replace the operator identifiers and attribute data of the connection operators with the operator identifiers and attribute data of the current operators not supported by the specified framework, and repeat this operation until all such operators are replaced.

[0147] Figure 9This is a partial flow chart of a computational graph processing method according to another embodiment of the present disclosure. As shown in the figure, the computational graph processing method specifically includes steps S901-S910. Steps S901-S905 are the same as steps S801-S805 and will not be repeated here.

[0148] Step S906 converts the computation graph into a json file.

[0149] Step S907 reads the connection operators from the json file in sequence.

[0150] Step S908: For the current connection operator, the operator identifier and attribute data of the first operator are read from the attribute data, and the operator identifier and attribute data of the connection operator are replaced respectively.

[0151] Step S909 determines whether all connection operators have been processed. If not, jump to step S907. If yes, execute step S910.

[0152] Step S910 converts the json file into a computation graph.

[0153] JSON (JavaScript Object Notation) files are a lightweight data exchange format that's easy for humans to read and write, as well as for machines to parse and generate. Therefore, in this implementation, the computation graph is converted to a JSON file to process the connection operators. The processed JSON file is then reloaded as a computation graph and returned to the specified deep learning framework. However, it should be noted that some frameworks do not support this conversion operation, so this conversion is only performed in frameworks that support it, such as the MxNet framework.

[0154] The above embodiment is further explained below using an example. For example, assuming the framework is the MxNet framework, the structure of an operator BatchNorm expressed in JSON consists of a key-value pair consisting of a key and a value. Among them, the key "op" corresponds to the operator identifier "BatchNo rm", the key "name" corresponds to the operator operation "resnetv10_stage1_batchnorm1_fwd", the key "attrs" corresponds to the attribute data, which includes multiple key values ​​composed of key and value, namely {"axis":"1","eps":"9.999999747378752e-06","fix_gam ma":"False","momentum":"0.89999999761581421","use_global_stats":"False"}, and the key i acceleration unit ts corresponds to the array [[23,0,0],[24,0,0], [25,0,0],[26,0,1],[27,0,1]], which is used to indicate the connection relationship of the operator BatchNorm.

[0155] The connection operator, converted from the JSON structure of BatchNorm, also consists of key-value pairs. The key "op" corresponds to the operator identifier "connection," the key "name" corresponds to the operator operation, which can be set to "resnetv10_stage1_batchnorm1_fwd," and the key "iAcceleratorUnits ts" corresponds to the array "[[23,0,0],[24,0,0],[25,0,0],[26,0,1],[27,0,1]]." The array corresponding to the key "iAcceleratorUnits ts" represents the connection relationship and contains the same data as BatchNorm, indicating that the two have the same connection relationship. In other words, this connection operator and the BatchNorm operator in the computation graph have the same keys "op," "name," and "iAcceleratorUnits ts."

[0156] In addition, the join operator also has an "attrs", but the structure of this "attrs" is different from the "attrs" in BatchNorm, as shown in the following table.

[0157] Table 2

[0158] property meaning Initial value i Acceleration unit t number Number of input tensors 1 output number Number of output tensors 1 op name Type of the original operator NULL op attribute List of properties of the original operator NULL

[0159] According to an embodiment of the present disclosure, corresponding to the above operator BatchNorm, the attribute data of "attrs" is:

[0160] "i Acceleration Unit t number":"3",

[0161] "output number":"2",

[0162] "op name":"BatchNorm",

[0163] "op attribute":"{"axis":"1","eps":"9.999999747378752e- 06","fix_gamma":"False","momentum":"0.8999999761581421","use_global_stats":"False"}",

[0164] }.

[0165] The italic part is that the attribute data corresponding to the key "attrs" is used as the attribute value of "op attrib ute".

[0166] In specific implementation, when constructing a connection operator, the operator connected to BatchNorm is used as an input parameter. Then, based on the number of input and output tensors and the input parameters, the same connection relationship as BatchNorm is established (i.e., the i acceleration unit ts data). The operator identifier "BatchNorm" is then stored as the value of "op name", and the attribute data corresponding to the operator "attrs" of BatchNorm is stored as the value of "op attribute". When replacing, the original operator identifier is taken from "op name", the value corresponding to "op" is replaced, and all data from "op attribute" is taken to replace the attribute data of "attrs" as a whole.

[0167] In this example, the number of input tensors and output tensor data of the current operator are used as attributes of the connection operator. Combined with the input parameters, the connection relationship of the connection operator is constructed. This ensures that the connection operator has the same connection relationship data as the current operator and its attribute data stores the attribute data and operator identifier of the current operator, allowing for replacement in subsequent steps. Using a JSON file to modify the operator type and attributes in the computation graph facilitates the replacement of connection operators using this file. Finally, the processed JSON file is reloaded into the computation graph.

[0168] The technical solution of the embodiment of the present disclosure can be applied to most frameworks such as TensorFlow framework, MxNet framework, Caffe framework, MxNet framework, etc., and therefore has certain versatility.

[0169] Furthermore, although the above description uses a data center server as an example of an execution subject of the present disclosure, the present disclosure is not limited thereto. Theoretically, the execution subject of the present disclosure can be any computing device, including the servers and terminal devices described above. For terminal devices, as long as the processor, memory, and network throughput capabilities of the terminal device can meet the operating requirements of the deep learning model, the deep learning model can be deployed thereon and various computational graph processing (including the computational graph processing solutions provided in the embodiments of the present disclosure) can be performed thereon.

[0170] The commercial value of the disclosed embodiments

[0171] Deep learning models currently have a wide range of successful applications, making even small improvements crucial not only from a technical perspective but also from a commercial perspective. For example, in the field of facial recognition, facial images can be identified using deep learning models from video surveillance cameras. By comparing these images with faces stored in the cloud, criminals can be identified in the surveillance footage. Similarly, in the field of speech recognition, deep learning models can be used to perform speech recognition and enable simultaneous interpretation. These applications can generate significant commercial benefits.

[0172] In the engineering practice of deep learning models, it is necessary to convert computational graphs under various frameworks into computational graphs adapted for a specific acceleration unit, and then return the computational graphs adapted for the specified acceleration unit to the original framework, thereby organically combining and mutually promoting the model's algorithm research and engineering application. The computational graph processing method provided in the disclosed embodiments helps reduce mapping functions, and therefore, the disclosed embodiments also have application prospects and commercial value.

[0173] Those skilled in the art will appreciate that the present disclosure can be implemented as systems, methods, and computer program products. Therefore, the present disclosure can be specifically implemented in the following forms, namely, complete hardware, complete software (including firmware, resident software, microcode), and can also be implemented in the form of a combination of software and hardware. In addition, in some embodiments, the present disclosure can also be implemented in the form of a computer program product in one or more computer-readable media, wherein the computer-readable medium contains computer-readable program code.

[0174] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium is, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by a processing unit, device or device, or used in combination with it.

[0175] A computer-readable signal medium may include a data signal propagated in baseband or as part of a chopped wave, which carries computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction system, apparatus, or device.

[0176] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0177] The computer program code for executing the disclosed embodiments can be written in one or more programming languages ​​or combinations thereof. The programming languages ​​include object-oriented programming languages, such as JAVA, C++, and can also include conventional procedural programming languages, such as C. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0178] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A processing unit, comprising: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit, configured to execute the decoded computer instructions to achieve: Converting a computational graph of a deep learning model used in a first deep learning framework into an intermediate expression conforming to the acceleration unit; performing model processing on the intermediate expression; Determining a first operator included in the processed intermediate expression and not registered in the first deep learning framework, wherein the processed intermediate expression represents the operator using an operator identifier, attribute data, and a connection relationship with other operators; Converting the processed intermediate expression back into a computational graph of the first deep learning framework includes: In the processed intermediate expression, the first operator is replaced with a join operator, and a join relationship between the join operator and other operators is constructed according to the join relationship between the first operator and other operators, and then the operator identifier and attribute data of the join operator are replaced with the operator identifier and attribute data of the first operator; The computation graph includes the connection relationship between operators; When constructing the connection operator, the operator identifier and attribute data of the first operator are stored as attribute data of the connection operator.

2. The processing unit according to claim 1, wherein: The operator identifier and attribute data of the first operator are stored in the attribute data of the join operator, and then replacing the operator identifier and attribute data of the join operator with the operator identifier and attribute data of the first operator includes: The operator identifier and attribute data of the first operator are read from the attribute data of the connection operator, and the operator identifier and attribute data of the connection operator are replaced respectively.

3. The processing unit according to claim 2, wherein: The step of replacing the first operator with the connection operator is repeated until all the first operators in the processed intermediate expression are replaced; then the step of replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator is repeated until all the connection operators in the processed intermediate expression are replaced.

4. The processing unit according to claim 1, wherein: The model processing includes at least one of the following: operator merging, quantization, graph slicing, and model pruning.

5. The processing unit according to claim 1 , wherein the instruction execution unit further implements: Before replacing the first operator with the join operator, converting the processed intermediate expression into a JSON file; and After replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator, converting the json file into a format of a computational graph followed by the first deep learning framework. The processing unit according to claim 1 , wherein: The first deep learning framework is an MxNet framework, and the connection operator is a registration operator of the MxNet framework.

7. The processing unit according to claim 1, wherein: The constructor of the connection operator specifies other operators that have a connection relationship with the first operator through input and / or output parameters, so that the constructor of the connection operator can construct the connection relationship between the connection operator and other operators according to the connection relationship between the first operator and other operators.

8. The processing unit according to claim 7, wherein: The constructor of the connection operator also stores the number of input tensors and the number of output tensors of the first operator in the attribute data of the connection operator.

9. The processing unit according to claim 1, wherein: The converting of the computation graph into an intermediate expression conforming to the acceleration unit includes: converting attribute data of at least one operator of the computation graph into attribute data of a corresponding operator defined by the intermediate expression through a mapping function.

10. The processing unit according to claim 1, wherein: The determining that the processed intermediate expression contains a first operator that has not been registered in the first deep learning framework includes: comparing the operator identifier of each operator in the processed intermediate expression with the operator identifier of the registered operator under the first deep learning framework to determine the first operator.

11. A computing device comprising a memory and the processing unit according to any one of claims 1 to 10. 12 . The computing device according to claim 11 , wherein the computing device is a server or a terminal device.

13. A method for processing a computational graph of a deep learning model, comprising: Converting a computational graph of a deep learning model used in a first deep learning framework into an intermediate expression conforming to the acceleration unit; performing model processing on the intermediate expression; Determining a first operator included in the processed intermediate expression and not registered in the first deep learning framework, wherein the processed intermediate expression represents the operator using an operator identifier, attribute data, and a connection relationship with other operators; Converting the processed intermediate expression back into a computational graph of the first deep learning framework includes: In the processed intermediate expression, the first operator is replaced with a join operator, and a join relationship between the join operator and other operators is constructed according to the join relationship between the first operator and other operators, and then the operator identifier and attribute data of the join operator are replaced with the operator identifier and attribute data of the first operator; The computation graph includes the connection relationship between operators; When constructing the connection operator, the operator identifier and attribute data of the first operator are stored as attribute data of the connection operator.

14. The computational graph processing method according to claim 13, wherein: The operator identifier and attribute data of the first operator are stored in the attribute data of the join operator, and then replacing the operator identifier and attribute data of the join operator with the operator identifier and attribute data of the first operator includes: The operator identifier and attribute data of the first operator are read from the attribute data of the connection operator, and the operator identifier and attribute data of the connection operator are replaced respectively.

15. The computational graph processing method according to claim 14, wherein: The step of replacing the first operator with the connection operator is repeated until all the first operators in the processed intermediate expression are replaced; then the step of replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator is repeated until all the connection operators in the processed intermediate expression are replaced.

16. The computational graph processing method according to claim 13, wherein: The model processing includes at least one of the following: operator merging, quantization, graph slicing, and model pruning.

17. The computational graph processing method according to claim 13, further comprising: Before replacing the first operator with the join operator, converting the processed intermediate expression into a JSON file; as well as After replacing the operator identifier and attribute data of the connection operator with the operator identifier and attribute data of the first operator, converting the json file into a format of a computational graph followed by the first deep learning framework.

18. The computational graph processing method according to claim 13, wherein: The first deep learning framework is an MxNet framework, and the connection operator is a registration operator of the MxNet framework.

19. The computational graph processing method according to claim 13, wherein: The constructor of the connection operator specifies other operators that have a connection relationship with the first operator through input and / or output parameters, so that the constructor of the connection operator can construct the connection relationship between the connection operator and other operators according to the connection relationship between the first operator and other operators.

20. The computational graph processing method according to claim 19, wherein: The constructor of the connection operator also stores the number of input tensors and the number of output tensors of the first operator in the attribute data of the connection operator.

21. A data center comprising the computing device according to claim 11 as a server.

Images