Tuning method, compilation method and computing device of deep learning model

Abstract

The present disclosure discloses a tuning method and a compilation method for a computing device and a deep learning model. The computing device includes a memory, a scheduling unit, and a designated acceleration unit. The memory stores instructions, and the processor reads instructions to execute: in the operator stream of the deep learning model, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to the input type and output type of the adjacent operators; the operator stream is deployed to the designated acceleration unit for execution. Inserting quantization and / or dequantization operators based on the input and output types of adjacent operators avoids inserting redundant quantization operations and / or dequantization operators, thereby ensuring that the instructions of the operator stream obtained are streamlined and improving the model efficiency and response speed of the deep learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of neural networks, and more specifically, to a tuning method, a compiling method, and a computing device for a deep learning model. Background Art

[0002] In recent years, with the continuous rise of artificial intelligence, deep learning models have become a widely used prediction model. As more and more applications are equipped with deep learning models, people are increasingly aware that although larger-sized deep learning models help improve the prediction accuracy of deep learning models, larger-sized deep learning models also occupy too much memory content and memory bandwidth during operation, thus affecting model efficiency and response speed.

[0003] To solve this problem, the current improvements to the size of deep learning models focus on two aspects: the first is to design a more effective network architecture to achieve acceptable prediction accuracy with a relatively small-sized deep learning model; the second is to reduce the size of the network through compression, encoding, etc. Quantization is one of the most widely used compression methods. Quantization converts floating-point operations into integer operations. Integer operations are many times faster than floating-point operations, so in theory the quantized deep learning model will be many times faster than the deep learning model before quantization. However, the methods in the prior art add quantization operators in some locations where quantization operations are not required, which makes the model efficiency of the quantized deep learning model worse than expected. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a tuning method, a compilation method and a computing device for a deep learning model to further improve the model efficiency of the quantized deep learning model.

[0005] The present disclosure provides a computing device, including a memory, a scheduling unit, and a designated acceleration unit, wherein the memory stores instructions, and the processor reads the instructions to execute:

[0006] In an operator stream of a deep learning model, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to input types and output types of the adjacent operators;

[0007] The operator flow is deployed to the designated acceleration unit for execution.

[0008] Optionally, inserting a quantization operator and / or a dequantization operator between adjacent operators according to input types and output types of adjacent operators includes:

[0009] When the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, a quantization operator and / or a dequantization operator is inserted between the adjacent operators.

[0010] Optionally, the input type and the output type are classified as a high-precision type and a low-precision type, and when the output type of the previous operator of the adjacent operator is different from the input type of the subsequent operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators includes:

[0011] When the output type of the previous operator of the adjacent operator is a high-precision type and the input type of the next operator is a low-precision type, a quantization operator is inserted; when the output type of the previous node is a low-precision type and the input operator of the next node is a high-precision type, a dequantization operator is inserted.

[0012] Optionally, when the output type of a preceding operator of the adjacent operator is different from the input type of a succeeding operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators includes:

[0013] When the output type of the previous operator of the adjacent operator is the first data type and the input type of the subsequent operator is the second data type, if the first data type and the second data type belong to the same precision type, an operator or a dequantization operator is inserted based on the bits occupied by the first data type and the second data type in the memory; if the first data type and the second data type do not belong to the same precision type, a quantization operator or a dequantization operator is inserted according to the precision type.

[0014] Optionally, inserting a quantization operator and / or a dequantization operator between the adjacent operators is also related to the input type and output type of the quantization operator and the dequantization operator.

[0015] Optionally, a predefined state machine is used to insert a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation between adjacent operators according to the input type and output type of the adjacent operators, and the state machine includes multiple state identifiers representing multiple data types and multiple directed connection edges connecting every two of the state identifiers and representing the input type and output type of each operator.

[0016] Optionally, the inserting a quantization operator and / or a dequantization operator between the adjacent operators according to the input type and the output type of the adjacent operators by using a predefined state machine includes:

[0017] Comparing each operator of the operator flow with a plurality of directed connection edges of the state machine to determine whether the directed connection edges are triggered;

[0018] When a directed connection edge is triggered, if the source state identifier of the triggered directed connection edge does not match the input type of the operator represented by the triggered directed connection, the quantization operator and / or the dequantization operator is inserted before the corresponding operator, or, if the target state identifier of the triggered directed connection edge does not match the output type of the operator represented by the triggered directed connection edge, the quantization operator and / or the dequantization operator is inserted after the corresponding operator.

[0019] Optionally, in the state machine, the data type represented by the initial state identifier is consistent with the input type of the first operator of the operator stream.

[0020] Optionally, in the state machine, the input type of the operator represented by the directed connection edge is the same as the source state identifier of the directed connection edge, and / or the output type of the operator represented by the directed connection edge is the same as the target state identifier of the directed connection edge.

[0021] Optionally, the method further includes: inserting the quantization operator at the beginning position of the operator stream, and inserting the dequantization operator at the end position of the operator stream.

[0022] Optionally, the input type and the output type are respectively one of the following types: 16-bit floating point number, 24-bit floating point number, 32-bit floating point number, 64-bit floating point number, 32-bit signed integer, 16-bit signed integer, 8-bit signed integer, 8-bit unsigned integer.

[0023] In a second aspect, an embodiment of the present disclosure provides a method for tuning a deep learning model, including:

[0024] In an operator stream of a deep learning model, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to input types and output types of the adjacent operators;

[0025] Deploy the operator flow to the specified acceleration unit for execution.

[0026] Optionally, inserting a quantization operator and / or a dequantization operator between adjacent operators according to input types and output types of adjacent operators includes:

[0027] When the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, it is determined to insert a quantization operator and / or a dequantization operator between the adjacent operators.

[0028] Optionally, the input type and the output type are classified as a high-precision type and a low-precision type, and when the output type of the previous operator of the adjacent operator is different from the input type of the subsequent operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators includes:

[0029] When the output type of the previous operator of the adjacent operator is a high-precision type and the input type of the next operator is a low-precision type, a quantization operator is inserted; when the output type of the previous node is a low-precision type and the input operator of the next node is a high-precision type, a dequantization operator is inserted.

[0030] Optionally, when the output type of a preceding operator of the adjacent operator is different from the input type of a succeeding operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators includes:

[0031] When the output type of the previous operator of the adjacent operator is the first data type and the input type of the subsequent operator is the second data type, if the first data type and the second data type belong to the same precision type, an operator or a dequantization operator is inserted based on the bits occupied by the first data type and the second data type in the memory; if the first data type and the second data type do not belong to the same precision type, a quantization operator or a dequantization operator is inserted according to the precision type.

[0032] Optionally, inserting a quantization operator and / or a dequantization operator between the adjacent operators is also related to the input type and output type of the quantization operator and the dequantization operator.

[0033] Optionally, the quantization operator and / or the dequantization operator is inserted between the adjacent operators according to the input type and output type of the adjacent operators using a predefined state machine, wherein the state machine includes multiple state identifiers representing multiple data types and multiple directed connection edges connecting every two of the state identifiers and representing the input type and output type of each operator.

[0034] Optionally, the using a predefined state machine to insert the quantization operator and / or the dequantization operator between the adjacent operators according to the input type and output type of the adjacent operators includes:

[0035] Comparing each operator of the operator flow with a plurality of directed connection edges of the state machine to determine whether the directed connection edges are triggered;

[0036] When a directed connection edge is triggered, if the source state identifier of the triggered directed connection edge does not match the input type of the operator represented by the triggered directed connection, the quantization operator and / or the dequantization operator is inserted before the corresponding operator, or, if the target state identifier of the triggered directed connection edge does not match the output type of the operator represented by the triggered directed connection edge, the quantization operator and / or the dequantization operator is inserted after the corresponding operator.

[0037] Optionally, in the state machine, the data type represented by the initial state identifier is consistent with the input type of the first operator of the operator stream.

[0038] Optionally, in the state machine, the input type of the operator represented by the directed connection edge is the same as the source state identifier of the directed connection edge, and / or the output type of the operator represented by the directed connection edge is the same as the target state identifier of the directed connection edge.

[0039] Optionally, the method further includes: inserting the quantization operator at the beginning position of the operator stream, and inserting the dequantization operator at the end position of the operator stream.

[0040] Optionally, the input type and the output type are respectively one of the following types: 16-bit floating point number, 24-bit floating point number, 32-bit floating point number, 64-bit floating point number, 32-bit signed integer, 16-bit signed integer, 8-bit signed integer, 8-bit unsigned integer.

[0041] In a third aspect, an embodiment of the present disclosure provides a method for compiling a deep learning model, wherein the method is implemented in a compiler and includes:

[0042] Compiling a deep learning model from a source code into a first instruction sequence;

[0043] Acquire an operator stream executable on a specified acceleration unit from the first instruction sequence;

[0044] In the operator stream, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to input types and output types of adjacent operators;

[0045] Deploy the processed operator stream to the specified acceleration unit for execution.

[0046] Optionally, the first instruction sequence is an intermediate instruction sequence generated by the compiler, and the second instruction sequence is an executable instruction sequence.

[0047] A fourth method, an embodiment of the present disclosure provides a data center, comprising any computing device as described above.

[0048] The tuning method of the deep learning model of the embodiment of the present disclosure determines whether to insert and execute quantization operators and / or dequantization operators between adjacent operators according to the input type and output type of the adjacent operators, so as to avoid inserting redundant quantization operators and / or dequantization operators, thereby ensuring the instruction simplification of the operator stream obtained thereby, and thereby improving the model efficiency and response speed of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0050] Figure 1 A hierarchical structure diagram of a data center to which an embodiment of the present disclosure is applied is shown;

[0051] Figure 2 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0052] Figure 3 is a block diagram of the internal structure of a server in a data center according to an embodiment of the present disclosure;

[0053] Figure 4 It is a control relationship diagram of a central processing unit (CPU) and a neural network acceleration unit (NPU) inside a server according to an embodiment of the present disclosure;

[0054] Figure 5 is an internal structure diagram of an NPU core according to an embodiment of the present disclosure;

[0055] Figure 6 is an architectural diagram of an exemplary software platform for compilation and optimization of deep learning models;

[0056] Fig. 7A and Figure 7B They are the operator flows before and after the quantization operator is inserted;

[0057] Figure 8 is a flow chart of a method for tuning a deep learning model provided by an embodiment of the present disclosure;

[0058] Fig. 9 is an exemplary state machine;

[0059] Fig.10 is an exemplary sequence of instructions that are transitioned through a state machine;

[0060] Fig.11 A comparison of a target operator stream obtained by processing according to the prior art and a target operator stream obtained by using an embodiment of the present disclosure based on the same original operator stream is shown;

[0061] Fig.12 It is a flowchart of a method for compiling a deep learning model provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0063] The following terms are used in this article.

[0064] Acceleration unit: also known as neural network acceleration unit, it is a processing unit designed to improve the data processing speed in some special-purpose fields (for example, processing images, processing various operations of neural networks, etc.) where the general-purpose processor is inefficient. It is often used in conjunction with a general-purpose processor CPU, and is controlled by the general-purpose processor to perform processing for specific purposes or specific fields, thereby improving computer processing efficiency in specific purposes or specific fields.

[0065] On-chip memory: Memory that is used exclusively within the main core or sub-core and cannot be shared.

[0066] Command Processor: The command interface between the accelerator unit and the central processing unit that drives the accelerator unit. The command processor receives instructions from the central processing unit for the accelerator unit to execute, and distributes these instructions to the various cores in the accelerator unit for execution. In addition, it is also responsible for the synchronization of the various cores in the accelerator unit.

[0067] Life cycle: An operand is not involved in all the processes in the instruction sequence. The part between its first appearance and the last use in the instruction sequence is the life cycle of the operand. In other words, after the life cycle, it is no longer used and there is no need to keep it in the on-chip memory.

[0068] Neural network: generally refers to artificial neural network (ANN), which is an algorithmic network that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. A classic neural network, which is also the simplest neural network structure, contains three levels: input layer, output layer and middle layer (also called hidden layer). The input layer, output layer and middle layer each include multiple nodes. A node is the smallest processing unit in a neural network. A very complex neural network structure can be formed through the extensive interconnection of a large number of simple function nodes.

[0069] Neural network model: In a neural network, nodes are mathematized to generate mathematical models of nodes. The mathematical models of a large number of nodes in the neural network constitute the neural network model.

[0070] Deep learning model: The concept of deep learning originates from the study of neural networks, and neural networks with multiple intermediate layers are called deep learning networks. Therefore, in this sense, deep learning models are also a kind of neural network models. Both deep learning models and neural network models must be generated through training. The sample data is input into the designed network structure (that is, the network structure has been determined), feature information is extracted through multiple intermediate layers, and the weight parameters of neurons are continuously corrected based on the output results of the output layer, so that the output results of the output layer tend to the preset results more and more, until the final weight parameters are determined. The trained deep learning model can be truly applied to actual scenarios, and the use of the deep learning model in actual scenarios can also be collected, which in turn optimizes the deep learning model.

[0071] Operator: refers to a collection of operations built in a deep learning model to achieve specific functions. A deep learning model can contain multiple such operators, and these operators can be executed on the hardware execution unit on the acceleration unit.

[0072] Figure 1 A hierarchical structure diagram of a data center is shown as a scenario in which an embodiment of the present disclosure is applied.

[0073] Data centers are a globally collaborative network of specific devices used to transmit, accelerate, display, calculate, and store data information on the Internet network infrastructure. In the future, data centers will also become competitive assets for enterprises. With the widespread application of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. As an important technology of artificial intelligence, neural networks have been widely used in data center big data analysis and calculation.

[0074] In traditional large data centers, the network structure is usually Figure 1 The three-layer structure shown is a hierarchical inter-networking model. This model includes the following three layers:

[0075] Access Layer 103: Sometimes also called the edge layer, including access switches 130 and servers 140 connected to the access switches. Each server 140 is a processing and storage entity in the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140. Access switches 130 are used to connect these servers to the switches in the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are usually located at the top of the rack, so they are also called top of rack switches, which are physically connected to the servers.

[0076] Aggregation Layer 102: sometimes also called distribution layer, including aggregation switches 120. Each aggregation switch 120 connects to multiple access switches and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0077] Core Layer 101: includes core switch 110. Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for multiple aggregation layers. The network of the entire data center is divided into L3 layer routing network and L2 layer routing network. Core switch 110 generally provides a flexible L3 layer routing network for the network of the entire data center.

[0078] Generally, the aggregation switch 120 is the demarcation point between the L2 and L3 layer routing networks. The network below the aggregation switch 120 is the L2 network, and the network above is the L3 network. Each group of aggregation switches manages a delivery point (POD), and each POD has an independent VLAN network. The server does not need to modify the IP address and default gateway when migrating within the POD, because one POD corresponds to one L2 broadcast domain.

[0079] The spanning tree protocol (STP) is usually used between the aggregation switch 120 and the access switch 130. STP makes only one aggregation switch 120 available for a VLAN network, and other aggregation switches 120 are used only when a failure occurs (dashed lines in the above figure). In other words, horizontal expansion cannot be achieved at the aggregation layer, because even if multiple aggregation switches 120 are added, only one is still working.

[0080] Figure 2 Shows Figure 1 The physical connection of the various components in a hierarchical data center. Figure 2As shown, one core switch 110 is connected to multiple aggregation switches 120 , one aggregation switch 120 is connected to multiple access switches 130 , and one access switch 130 is connected to multiple servers 140 .

[0081] server

[0082] Since the server 140 is the real computing device in the data center, Figure 3 The structure block diagram of the internal structure of a server 140 is shown. The server 140 includes a memory 210, a central processing unit (CPU) 220 and various acceleration units connected by a bus. These acceleration units include a neural network acceleration unit (NPU) 230, a data transfer unit (DTU) 260, a graphics processing unit (GPU, not shown), an application-specific integrated circuit (ASIC, not shown) and a field programmable gate array (FPGA, not shown).

[0083] The architecture design of traditional processors makes the control unit and storage unit occupy a large part of the space in the architecture, while the computing unit occupies insufficient space. Therefore, it is very effective in logic control, but not efficient enough in large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively improve the computing speed for calculations of different functions and different fields. The acceleration unit proposed in the present disclosure can be any of them, and these acceleration units are introduced below.

[0084] Neural network acceleration unit (NPU) 230: It is a processing unit that uses a data-driven parallel computing architecture to process a large number of operations (such as convolution, pooling, etc.) of each neural network node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each neural network node are closely related in the entire computing process and will be used frequently, using the existing CPU architecture, since the memory capacity in the CPU core is very small, a large number of external core memories must be accessed frequently, resulting in inefficient processing. With the use of NPU, each core has an on-chip memory with a storage capacity suitable for neural network calculations, avoiding frequent access to the memory outside the core, which can greatly improve processing efficiency and computing performance.

[0085] Data transfer unit (DTU) 260: It is a wireless terminal device specially used to convert serial port data into IP data or convert IP data into serial port data for transmission through wireless communication network. The main function of DTU is to transmit the data of remote device back to the background center wirelessly. At the front end, DTU and the customer's device are connected through an interface. After DTU is powered on, it first registers to the mobile GPRS network, and then establishes a socket connection with the background center set in DTU. The background center serves as the server of the socket connection, and DTU is the client of the socket connection. Therefore, DTU and background software are used together. After the connection is established, the front-end device and the background center can perform wireless data transmission through DTU.

[0086] Graphics Processing Unit (GPU): A microprocessor that specializes in image and graphics-related computing. GPUs have exploited the shortcoming of too little space for computing units in CPUs, and use a large number of computing units dedicated to graphics computing, which reduces the graphics card's dependence on the CPU and takes on some of the computationally intensive graphics and image processing tasks that the CPU originally undertook.

[0087] Application-specific integrated circuit (ASIC): refers to an integrated circuit designed and manufactured to meet the needs of specific users and specific electronic systems. Since this integrated circuit is customized according to user requirements, its structure is often adapted to specific user requirements.

[0088] Field Programmable Gate Array (FPGA): It is a product further developed on the basis of programmable devices such as PAL and GAL. It appears as a semi-custom circuit in the field of Application Specific Integrated Circuit (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0089] Although the acceleration unit has the advantage of much higher execution efficiency than ordinary processors for specific applications or fields, it is also subject to the control of the scheduling unit 220. Taking the NPU as an example, the memory 210 stores various deep learning models, including the neurons and weight data of the neurons in these models. These deep learning models are called when needed. Figure 3A scheduling unit 220 is deployed to an acceleration unit 230. Specifically, the scheduling unit 220 can inform the acceleration unit 230 of the storage location of the deep learning model in the memory 210 in the form of instructions. The acceleration unit 230 can then address these locations and store the instructions to be executed in its on-chip memory. The scheduling unit 220 can also send the instructions to be executed of the acceleration unit 230 to the acceleration unit 230 in the form of instructions, and the acceleration unit 230 receives the instructions and stores them in the on-chip memory. Similarly, the acceleration unit 230 can also obtain input data in the above manner. The acceleration unit 230 obtains the instructions to be executed and the input data for inference calculation. The weight parameters of the node can be included in the instruction sequence of the deep learning model, and the acceleration unit 230 is taken out from the memory 210. Of course, the weight parameters of the node can also be stored independently, and the acceleration unit 230 can take them out from the memory 210 when needed. Here, the scheduling unit 220 can be understood as a hardware unit with scheduling and control capabilities, which can generally be a hardware unit such as a central processing unit (CPU), a microcontroller, or a microprocessor.

[0090] Internal structure of the scheduling unit and acceleration unit

[0091] Combine the following Figure 4 The internal structure diagram of the scheduling unit and the acceleration unit specifically explains how the scheduling unit controls the acceleration unit to work.

[0092] like Figure 4 As shown, the scheduling unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0093] The instruction fetch unit 223 is used to transfer the instruction to be executed from the memory 210 to the instruction register (which may be Figure 4 A register in the register file 229 shown is used to store instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0094] After fetching the instruction, the dispatch unit 220 enters the instruction decoding stage, and the instruction decoding unit 224 decodes the fetched instruction according to the predetermined instruction format to obtain the operand acquisition information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand acquisition information, for example, points to an immediate number, a register, or other software / hardware that can provide a source operand.

[0095] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions so as to efficiently distribute each instruction to different instruction executing units 226 , thereby making it possible to operate multiple instructions in parallel.

[0096] After the instruction transmitting unit 225 transmits the instruction to the instruction executing unit 226, the instruction executing unit 226 starts to execute the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by the acceleration unit, it forwards it to the corresponding acceleration unit for execution. For example, if the instruction is an instruction for neural network inference, the instruction executing unit 226 no longer executes the instruction, but sends the instruction to the acceleration unit 230 via the bus, and the acceleration unit 230 executes it.

[0097] The acceleration unit 230 includes multiple cores 236 ( Figure 4 Four cores are shown, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0098] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus. According to different mechanisms, the bus channel 231 may include a PCIE channel 232, an I2C channel 233, and a JTAG channel 234.

[0099] PCIE, or PCI-Express, is a high-speed serial computer expansion bus standard proposed by Intel in 2001 to replace the old PCI, PCI-X and AGP bus standards. PCIE is a high-speed serial point-to-point dual-channel high-bandwidth transmission. The connected devices are allocated exclusive channel bandwidth and do not share bus bandwidth. It mainly supports active power management, error reporting, end-to-end reliability transmission, hot plugging and quality of service and other functions. Its main advantage is its high data transmission rate, and it also has considerable development potential. At present, most PCIE buses are PCIE GEN3, but the disclosed embodiment can also use PCIE GEN4, that is, a bus channel that complies with the PCI-Express4.0 standard.

[0100] I2C channel 233 is a simple, bidirectional, two-wire synchronous serial bus channel developed by Philips. It only requires two wires to transmit information between devices connected to the bus.

[0101] JTAG is the abbreviation of Joint Test Action Group, which is the common name of the IEEE standard 1149.1 called Standard Test Access Port and Boundary Scan Architecture. This standard is used to verify the functionality of printed circuit boards produced by design and testing. In 1990, JTAG was formally standardized by IEEE document 1149.1-1990, and in 1994, a supplementary document was added to describe the boundary scan description language (BSDL). Since then, this standard has been widely adopted by electronic companies around the world. Boundary scan has almost become synonymous with JTAG. JTAG channel 234 is a bus channel that follows this standard.

[0102] The direct memory access (DMA) mechanism 235 is a function provided by some computer bus architectures, which enables data to be written directly from an attached device (such as an external memory) to the on-chip memory of the acceleration unit 230. Compared with the method in which all data transmissions between devices must pass through the command processor 237, this method greatly improves the efficiency of data access. It is precisely because of this mechanism that the core of the acceleration unit 230 can directly access the memory 210, read the parameters in the deep learning model (such as the weight parameters of each node), etc., which greatly improves the efficiency of data access. Although the direct memory access mechanism 235 is shown in the figure as being located between the processor 237 and the bus channel 231, the design of the acceleration unit 230 is not limited to this. In some hardware designs, each NPU core 236 can include a direct memory access mechanism 235, so that the NPU core 236 does not need to go through the command processor 237, but directly reads data from the attached device and writes it to the on-chip memory of the acceleration unit 230.

[0103] The command processor 237 assigns the instructions sent by the scheduling unit 220 to the acceleration unit 230 to the core 236 for execution. The instruction execution unit 226 sends the instructions to be executed by the acceleration unit 230 to the acceleration unit 230 or the instruction execution unit 226 informs the storage location of the instructions to be executed on the memory 210. After the sequence of instructions to be executed enters from the bus channel 231, it is cached in the command processor 237, and the command processor 237 selects the core 236 and assigns the instruction sequence to it for execution. The instructions to be executed come from the compiled deep learning model. It should be understood that the sequence of instructions to be executed may include instructions to be executed in the scheduling unit 220 and instructions to be executed in the acceleration unit 230.

[0104] NPU core

[0105] Figure 5 Detailed description of the internal structure of an NPU core according to an embodiment of the present disclosure.

[0106] In one embodiment, Figure 5 As shown, the NPU core 236 includes a tensor engine 310, a pooling engine 320, a convolution process 330, an activation operation 380, a sequencer 350, an instruction cache 340, an on-chip memory 360, and a constant buffer 370. Among them, the tensor engine 310, the pooling engine 320, the convolution process 330 and the activation operation 380 are all hardware execution units. The hardware execution unit is a hardware module in the acceleration unit that is actually used to perform various operations. Some hardware execution units are not shown in the figure.

[0107] The instruction sequence assigned by the command processor 237 to the NPU core 236 first enters the instruction cache 340 cache. Then, the sequencer 350 fetches instructions from the instruction cache 340 in a first-in-first-out order and assigns them to each hardware execution unit for execution according to the nature of the instruction. The tensor engine 310 is responsible for processing tensor-related operations in the deep learning model. The pooling engine 320 is responsible for processing pooling operations in the deep learning model. The convolution processing 330 is responsible for the convolution operation in the deep learning model. The activation operation 380 is used to perform the operation corresponding to the activation function in the deep learning model. The sequencer 350 decides to assign the instruction to each hardware execution unit for execution based on the nature of the operation such as convolution, matrix multiplication or pooling.

[0108] The on-chip memory 360 is an in-core memory that stores weight parameters in the deep learning model and inputs and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters (e.g., hyperparameters in the deep learning model) in addition to weight parameters in the deep learning model. As described above, in the process of the scheduling unit 220 pre-configuring the deep learning model in the acceleration unit 230, the scheduling unit 220 can send the location of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (e.g., hyperparameters). For weights, the acceleration unit 230 will take them out from the corresponding position of the memory 210 when needed and put them in the on-chip memory 360. For other parameters, the acceleration unit 230 will take them out from the corresponding position of the memory 210 when needed and put them in the constant buffer 370. In addition, when the executable instruction is assigned to the core 236 for execution by the command processor 237, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, various intermediate results obtained are also stored in the on-chip memory 360 .

[0109] Software platform for deep learning models

[0110] The improvement of deep learning models requires not only the support of the above hardware layer, but also the continuous improvement of the software layer and algorithm layer. Only by combining the best hardware support at the bottom layer and the best deep learning algorithm structure above can a powerful computing engine be delivered.

[0111] Figure 6 The software platform of an exemplary deep learning model is shown. The software platform is deployed on the terminal device of the R&D personnel, and provides various environments suitable for the deep learning model, including the editing environment, the compiling environment, the quantization operation environment, and the simulation operation environment. The R&D personnel can implement a series of operations such as code editing, code compilation, parameter adjustment, quantization, simulation operation (i.e., running the model on the CPU) of the deep learning model based on the platform.

[0112] As shown in the figure, from top to bottom, the architecture diagram includes an application layer 401, a framework layer 402, and a function layer 403.

[0113] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, recommendation 407, etc. These applications are built using this architecture, and can also call the architecture to provide a running interface in the application to obtain reasoning capabilities in the application.

[0114] The framework layer 402 integrates open source platforms such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to continue to optimize and improve various algorithms. TensorFlow 408 is a symbolic mathematical system based on data flow programming, which is widely used in the programming implementation of various machine learning algorithms. MXNet 409 is the deep learning library selected by Amazon. Caffe 410, the full name of Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modular thinking.

[0115] The functional layer 403 includes a compilation stack 403 and a running stack 404. The compilation stack 403 is used to convert (converter) 411, quantize (quantization) 412, optimize (optimization) 413 and compile (compile) 414 various models. Conversion 411 is to provide the internal data of the model to be converted into an intermediate (IR) format. Quantization 412 is to convert the weight parameters in the deep learning model and the input parameters to the deep learning model from high-precision types to low-precision types. Optimization 413 includes operations such as fusion of operators inside the model and multi-model optimization linking. Compilation 414 is to optimize the model according to the acceleration unit to generate a binary model that can be recognized by the hardware. The running stack 404 includes a running API 415, an execution manager 416, a user mode driver 417 and a kernel mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized running API 415 is used to provide various interfaces that can be called at runtime. The user mode driver 417 is used to provide hardware commands and resource scheduling in kernel mode. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0116] It should be understood that Figure 6 What is shown is a general software platform. There are also some software platforms on the market that are dedicated to building specific types of deep learning models. Of course, the embodiments of the present disclosure do not rely on a specific software platform.

[0117] Quantization and Dequantization Operations

[0118] The quantization operation described in the present disclosure is to convert the input data of the deep learning model from high-precision data to low-precision data, and the dequantization operation is to convert the input data of the deep learning model from low-precision data to high-precision data. Generally speaking, we divide floating-point data into high-precision data and integer data into low-precision data, then the conversion from high-precision data to low-precision data is a quantization operation, and vice versa. If the two data are of the same precision, the precision type is determined according to the bits occupied by each of the two data. The data types that can be used in the current deep learning model include but are not limited to the following: 16-bit floating point numbers, 24-bit floating point numbers, 32-bit floating point numbers (fp32), 64-bit floating point numbers (fp64), 32-bit signed integers (int32), 16-bit signed integers (int16), 8-bit signed integers (int8), 8-bit unsigned integers (uint8). For example, converting 32-bit floating point data to 8-bit signed integer data requires a quantization operation, and vice versa. For another example, converting 32-bit floating point data into 16-bit floating point data requires performing a quantization operation, and vice versa, performing a dequantization operation.

[0119] In specific implementation, quantization operations and inverse quantization operations can also be divided into linear mapping methods and nonlinear mapping methods. In the case of linear quantization, quantization can be performed using the following formula. δ is a scaling factor, which is equivalent to the value after quantization per unit representing the value difference before quantization. Considering the zero point problem (error-free representation of floating point numbers as 0), it is necessary to add an offset to the quantization method, namely zq. δ and zq are determined by the maximum value xmax and minimum value xmin of the input to be quantized. It quantizes the input to be quantized within [xmin, xmax] to any integer between 0 and n-1 (including 0 and n-1). Let q(x) be the value quantized to the input to be quantized x, and there is a formula:

[0120]

[0121] Where clamp is the limit function, and formula 1 means round((xx min ) / δ) is limited to between 0 and n-1. If round((xx min ) / δ) is greater than n-1, then the value of formula 1 is n-1; if round((xx min ) / δ) is less than 0, then the value of formula 1 is 0; if round((xx min ) / δ) is between 0 and n-1, then the value of Formula 1 is round((xx min ) / δ).

[0122] The scaling factor δ after taking into account the zero point problem is calculated as follows:

[0123]

[0124] Due to the zero point problem, we cannot simply use the value of (x / δ) to determine which of 0 to n-1 it is quantized to. We must consider the offset z q The offset z q It is calculated as follows:

[0125]

[0126] Considering the offset z q Then, the quantized value q(x) is determined according to the pre-quantized value x according to the following formula:

[0127]

[0128] It should be noted that the above only provides one quantization method, and those skilled in the art know that there are other quantization methods. In other quantization methods, there are other quantization formulas.

[0129] Now combined Fig. 7A and Figure 7B The example of Fig. 7A In FIG. 7 , sequence 701 is an operator stream of a deep learning model, which is composed of operators that can be run on a specified acceleration unit, and represents a deep learning model before the quantization operator is inserted (only a part of it). Sequence 701 includes a convolution (conv) operator, a pooling (pooling) operator, and another convolution (conv) operator.

[0130] In the acceleration unit, since the input and output types of each hardware execution unit are fixed, it is necessary to pay attention to the high-precision and low-precision state conversion of the input and output data of the deep learning model according to the requirements of calculation and access. That is, sometimes high-precision data needs to be converted into low-precision data, and sometimes low-precision data needs to be converted into high-precision data.

[0131] Figure 7B FIG. 4 shows a deep learning model after inserting a quantization operator according to the prior art. Figure 7B In the figure, sequence 702 is the operator stream of the deep learning model, which represents the deep learning model after the quantization operator is inserted (only a part of it). Sequence 702 includes a convolution (conv) operator, a dequantization operator (DQ), a quantization operator (Q), a pooling operator (pooling), a dequantization operator (DQ), a quantization operator (Q), a convolution operator (conv), a dequantization operator (DQ), and a quantization operator (Q).

[0132] Deep learning model Figure 6 After the software platform shown is compiled, an instruction sequence that can be executed on the specified acceleration unit is obtained. The scheduling unit 220 controls the deployment of the operator stream that can be executed on the acceleration unit to the acceleration unit 230, and the instruction sequence that cannot be executed by the acceleration unit 230 is deployed to the scheduling unit 220 for execution. After the acceleration unit 230 reads the executable operator stream from the memory 210, it distributes it to multiple NPU cores 236 for execution. The NPU core 236 includes multiple hardware execution units. Each hardware execution unit can execute one or more operators. The executable operators are assigned to the corresponding hardware execution units by the sequencer 350 in the NPU core 236. For example, the sequence 702 is Figure 6 After the software platform shown is compiled, it is converted into an operator stream that can be executed on the acceleration unit 230. The acceleration unit 230 distributes the operator stream to each hardware execution unit: the instructions corresponding to the pooling operator are distributed by the sequencer 350 to the pooling engine 320 for execution, the convolution operator is distributed by the sequencer 350 to the convolution processing 330 for execution, the quantization operator will be distributed to the quantization execution unit for execution, and the dequantization operator will be distributed to the dequantization execution unit for execution, so as to complete the quantization operation and the dequantization operation.

[0133] In the prior art, whether to insert a quantization or dequantization operator is determined based on the input and output type of a single operator. Figure 7B In the sequence 702 shown, the pooling operator is first looked at. The output type of the pooling operator is an 8-bit signed integer, which is converted into a quantized value. Then the convolution operator is looked at. The input type of the convolution operator is also an 8-bit signed integer, so an inverse quantization operator is inserted.

[0134] The tuning solution disclosed in this disclosure

[0135] Figure 8 The present invention is a flowchart of a method for tuning a deep learning model according to an embodiment of the present disclosure. The quantization method is applied at the algorithm level and involves improving the source code of the deep learning model. The method specifically includes the following steps.

[0136] Step S801: In the operator flow of the deep learning model, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to the input type and output type of the adjacent operators.

[0137] Step S802: deploy the operator flow to the designated acceleration unit for execution.

[0138] First, the operator stream of the deep learning model is obtained. The operator stream can be obtained based on the source code or executable code of the deep learning model. Each operator contained in the operator stream can be executed on a specified acceleration unit, that is, each operator in the operator stream has a corresponding hardware execution unit on the specified acceleration unit. We know that the acceleration unit is specially designed for artificial intelligence neural networks to accelerate the operation of neural networks and solve the problem of low efficiency of traditional chips in neural network operations. Many hardware execution units are built on the acceleration unit, and these hardware execution units have high execution efficiency when executing instructions. Accordingly, in the case of Figure 6 The framework layer shown builds multiple operators for the deep learning model based on various hardware execution units on the specified acceleration unit. These operators can be executed on the corresponding hardware execution units, and the input and output types of each operator are consistent with the input and output types of the corresponding hardware execution units. For example, convolution operators, pooling operators, quantization operators, dequantization operators, etc. are built in the framework layer.

[0139] Then determine the input and output types of each operator in the operator stream. The input and output types of each operator of the deep learning model can be stored in Table 1. Table 1 lists an exemplary input and output type of each operator. As shown in the figure, for example, the input and output type of the convolution operator is a low-precision data type of an 8-bit signed integer, and the input and output type of the pooling operator is also a low-precision data type of an 8-bit signed integer. Based on Table 1, the input and output type of each operator in the above operator stream can be obtained.

[0140] Table 1

[0141]

[0142]

[0143] Then, according to the input type and output type of the adjacent operators, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between the adjacent operators. Based on Table 1, take the adjacent convolution operator and activation operator as an example. The input of the convolution operator is an 8-bit signed integer, the output type is a 32-bit signed integer, and the input and output types of the activation operator are also 24-bit floating point numbers. Therefore, a dequantization operator needs to be inserted between the convolution operator and the activation operator to convert the output data of the convolution operator from a 32-bit signed integer to a 24-bit floating point number. For another example, based on Table 1, take the adjacent convolution operator and pooling operator as an example. Since the input of the convolution operator is an 8-bit signed integer, the output type is a 32-bit signed integer, and the input and output types of the pooling operator are both 8-bit signed integers, it is necessary to first insert a dequantization operator between the convolution operator and the pooling operator, and then insert a quantization operator.

[0144] Finally, the operator stream is deployed to the designated acceleration unit for execution, that is, the operator stream into which the quantization operator and / or the dequantization operator is inserted is deployed to the designated acceleration unit for execution.

[0145] In summary, this embodiment determines whether to insert and execute quantization operators and / or dequantization operators between adjacent operators according to the input and output types of each operator, so as to avoid inserting redundant quantization operators and / or dequantization operators, thereby ensuring the instruction conciseness of the operator stream obtained, thereby improving the model efficiency and response speed of the deep learning model.

[0146] In some embodiments, the input and output types of adjacent operators refer to the output type of the previous operator and the input type of the next operator of the adjacent operator. If the output type of the previous operator is the same as the input type of the next operator, it is determined not to add a quantization operator and / or a dequantization operator before the adjacent operator. When the output type of the previous operator of the adjacent operator is different from the input type of the next operator of the adjacent operator, it is determined to add a quantization operator and / or a dequantization operator between the adjacent operators. The operator stream after the above processing can be transmitted through the following method: Figure 6 After the software platform shown is compiled, it is deployed to the specified acceleration unit for execution.

[0147] In some embodiments, the input types and output types of various operators can be classified into various precision types, such as high-precision types or low-precision types. If the output type of the previous operator of an adjacent operator is a low-precision type and the input type of the next operator is a high-precision type, an inverse quantization operator is inserted between the two operators. If the previous operator of an adjacent operator is a high-precision type and the input type of the next operator is a low-precision type, a quantization operator is inserted between the two operators. If the output type of the previous operator and the input type of the next operator belong to the same precision, no insertion is required.

[0148] In some embodiments, the output type of the current operator is the first data type, and the input type of the next operator is the second data type. If the first data type and the second data type belong to the same precision type, then the insertion is determined based on the bits occupied by the first data type and the second data type in the memory. Whether the quantization operator or the dequantization operator is inserted, if the first data type and the second data type do not belong to the same precision type, then the insertion is determined according to the precision type. The first data type and the second data type are one of the following data types: 16-bit floating point number, 24-bit floating point number, 32-bit floating point number (fp32), 64-bit floating point number (fp64), 32-bit signed integer (int32), 16-bit signed integer (int16), 8-bit signed integer (int8), 8-bit unsigned integer (uint8). The difference between this embodiment and the previous embodiment is that the previous embodiment does not need to convert data under the same precision type. The embodiment method converts from any data type to another data type, and quantization and / or dequantization operations are required. For example, from int32 to int8, quantization operations also need to be performed.

[0149] In some embodiments, the input and output types of each operator can be stored in a database table. When the operator stream of the deep learning model is obtained, the database table is retrieved according to the identifier of each operator to obtain the input and output type of each operator, and then it is determined whether to insert quantization and / or dequantization operators between adjacent operators based on the input and output types of adjacent operators.

[0150] In some embodiments, a predefined state machine is used to determine whether to insert a quantization operator and a dequantization operator. The state machine includes a plurality of state identifiers representing a plurality of data types and a plurality of directed connection edges connecting every two of the state identifiers and representing the input type and output type of each operator. Fig. 9 The state machine 10 is shown for illustration.

[0151] As shown in the figure, the start symbol indicates the initial state and represents the data type of Int8. State 1 is an intermediate state and represents the data type of int32. State 2 is an intermediate state and represents the data type of fp24. There are two directed edges ① and ③ between the start state and state 1. The direction of the arrow of the directed edge ① indicates that the start symbol is the source state and state 1 is the target state. The edge of the directed edge ① represents the convolution operator, and the input and output types of the convolution operator are marked as int8 and int32 on the figure. The direction of the arrow of the directed edge ③ indicates that state 1 is the source state and the start symbol is the target state. The edge of the directed edge ③ represents any operator except the activation operator Relu. There is a directed edge ② between state 1 and state 2. The direction of the arrow of the directed edge ② indicates that state 1 is the source state and state 2 is the target state. The edge of the directed edge ② represents the activation operator, and the input and output types of the activation operator are marked as fp24 on the figure. There is a directed edge ④ between state 2 and the start symbol. The direction of the arrow of the directed connection edge ④ indicates that state 2 is the source state, the start mark is the target state, and the connection edge of the directed connection edge ④ represents an arbitrary operator.

[0152] The following describes how to use the state machine. Define the data type of state identifier i as Si. The transition between two state identifiers is triggered by the comparison result between the operator in the operator stream and the node on the graph. Define the input type Ti and output type To of each operator on the graph, such as Ti=int8 and To=int32 for the convolution operator conv; Ti=To=f24 for the activation operator relu. For any operator stream, starting from the start identifier, the input data type must be int8. This is because the operator stream of a deep learning model usually starts with the convolution operator. Since the input type of the convolution operator is int8, it is necessary to ensure that the input type of the operator stream is also int8. If the input data of the operator stream is not int8, you can also insert a dequantization operator before the operator stream to obtain int8 input data. Compare each operator of the operator stream with the operators represented by the directed connection edges of the state machine in order. When the operator causes the state to transfer, the state will change from i->j. First check whether Si and Ti of the current state are the same. If they are different, insert the corresponding quantization and / or dequantization nodes (Q / DQ) to complete the conversion of Si->Ti. If the conditions are not met, return to the start state via failure. Each time the failure state changes, it is necessary to check whether Si and Sj (always start) are the same. If they are different, insert the corresponding quantization and dequantization nodes (Q / DQ) to complete the conversion.

[0153] Continue below Fig.10The sequence shown in FIG. 1 is used as an example to further explain the operation of the state machine. The operation of the state machine starts from the start identifier. First, look at the directed connection edge starting from start. There is only one directed connection edge ①, so compare the instruction 1 on the left with the operator represented by the directed connection edge ①. If they are the same, it means that instruction 1 causes the state of the state machine to change, that is, jump from the start state to state 1. Instruction 1 on the left is a convolution operator, which is exactly the same as the operator represented by the directed connection edge ①, so the state machine jumps from the start state to state 1. Then compare the start state (i.e., int8) with the input type of the convolution operator conv (i.e., int8) to determine that the two are the same, so there is no need to insert a quantization operator between start and the convolution operator conv (i.e., before the operator conv). Since in this example, the target state of each directed connection edge is set to be the same as the output type of the operator represented by the directed connection edge, there is no need to compare the target state of the directed connection edge with the output type of the operator represented by the directed connection edge.

[0154] At this time, due to instruction 1, the state machine is already in state 1. As can be seen from the figure, there are directed edges ② and ③ starting from state 1. Instruction 2 is the convolution operator conv. Compare instruction 2 with directed edge ② to determine that the operators represented by instruction 2 and directed edge ② are different. Directed edge ③ is a failure edge. Reset the state of the state machine to the start state to start a new round of state jumps, that is, the state machine needs to return from int32 to int8. At this time, it is necessary to insert the dequantization operator DQ and the quantization operator Q between state 1 and the start state (that is, between instructions 1 and 2). Use DQ to convert int32 to fp24, and Q to convert fp24 to int8.

[0155] At this point, the state machine returns to the start state. Continue to compare instruction 2 on the left with the operator represented by directed edge ①. If they are the same, it means that instruction 2 causes the state machine to change, that is, jump from the start state to state 1. Instruction 2 on the left is a convolution operator, which is exactly the same as the operator represented by directed edge ①, so the state machine jumps from the start state to state 1 again. Then compare the start state (i.e. int8) with the input type of the convolution operator conv (i.e. int8) to determine that the two are the same, so there is no need to insert a quantization operator between start and operator conv (i.e. before operator conv).

[0156] At this point, the state machine has reached state 1. Continue to compare instruction 3 on the left with the operator represented by directed edge ②. If they are the same, it means that instruction 3 causes the state machine to change. Instruction 3 on the left is the activation operator, which is exactly the same as the operator represented by directed edge ②, so the state machine jumps from state 1 to state 2 again. Then compare state 1 (i.e. int32) with the input type of the activation operator (i.e. int24) to determine that the two are different, so it is necessary to insert the dequantization operator DQ between states 1 and 2 (i.e. between instructions 2 and 3).

[0157] At this point, the state machine has reached state 2. Continue to compare instruction 4 on the left with the directed edge ④ starting from state 2. Instruction 4 on the left is a convolution operator. Directed edge ④ represents a failure edge, and the state machine needs to return to the start state to start the next round of state jumps. The state machine returns from fp32 to int8, so it is necessary to insert the quantization operator Q between the two states (between instruction 3 and state 4) because Q converts fp24 to int8.

[0158] At this point, the state machine returns to the start state. Continue to compare instruction 4 on the left with the operator represented by the directed connection edge ①. If they are the same, the state machine jumps to state 1. The quantized instruction sequence is shown in the sequence on the right of the figure.

[0159] It should be pointed out that usually after the operator stream ends, the data output by the operator stream at the end needs to be converted into a quantized value. Since the last operator of the above operator stream is a convolution operator with an output type of int32, it is necessary to insert a dequantization operator and a quantization operator at the end position of the above operator stream to convert the output data of the operator stream into a quantized value.

[0160] It should also be pointed out that the above state machine has some special features: 1) The output type of the operator represented by the directed connection edge is set to be the same as the target state of the directed connection edge, so that when performing a comparison, only the input type of the operator and the source state can be compared; 2) The data type represented by the initial state identifier is set to be consistent with the input type of the convolution operator, which is a quantized value; 3) The failure connection edge is set to return to the initial state identifier to start the next round of state jumps.

[0161] Of course, those skilled in the art can construct Fig.10 State machines with different structures can also implement the embodiments of the present disclosure. For example, considering the network structure of the deep learning model, it is entirely possible to construct a state machine with more state identifiers and more directed connection edges to further cover the possible jumps that may occur when the operator flow of the deep learning model is executed on a specified acceleration unit.

[0162] Fig.11A comparison of the target operator stream obtained after processing according to the prior art and the target operator stream obtained by using the embodiment of the present disclosure based on the same original operator stream is shown. As shown in the reference figure, the original operator stream is: convolution operator (conv) -> pooling operator (pooling) -> pooling operator (pooling). The left side shows that the target operator stream obtained after processing according to the prior art is: convolution operator (conv) -> dequantization operator (DQ) -> quantization operator (Q) -> pooling operator (pooling) -> dequantization operator (DQ) -> quantization operator (Q) -> convolution operator (conv) -> dequantization operator (DQ) -> quantization operator (Q). The right side shows that the target operator stream obtained by using the embodiment of the present disclosure is: convolution operator (conv) -> dequantization operator (DQ) -> quantization operator (Q) -> pooling operator (pooling) -> dequantization operator (DQ) -> quantization operator (Q) -> convolution operator (conv) -> dequantization operator (DQ) -> quantization operator (Q). By comparison, it can be seen that in the embodiment of the present disclosure, since the input and output types of adjacent operators are considered together, it is found that the output type of the pooling operator is int8, and the input type of the convolution operator is also int8. There is no need to insert a quantization operator and a dequantization operator between these two operators, thereby avoiding the insertion of redundant operators.

[0163] Fig.12 The compilation method of the deep learning model provided by the embodiment of the present disclosure is also applied to the algorithm level. More specifically, the compilation method is implemented by a deep learning compiler, which can be applied to Figure 6 In the compilation stack 403 shown in FIG.

[0164] Step S110, compile the deep learning model from the source code into a first instruction sequence. In this step, the source code of the deep learning model is compiled to obtain a first instruction sequence, which may be an executable instruction sequence or an intermediate instruction sequence generated during the compilation process.

[0165] Step S120, obtaining an operator stream executable on a designated acceleration unit from the first instruction sequence. Figure 4 The scheduling unit will execute part of the instructions in the first instruction sequence on the scheduling unit and execute the other part of the instructions on the designated acceleration unit. Therefore, this step will find and extract the instruction sequence that can be executed on the designated unit as an operator stream.

[0166] Step S130: In the operator stream, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to the input type and output type of the adjacent operators. The input and output types of each operator of the deep learning model are collected, and then the insertion of a quantization or dequantization operator between adjacent operators is determined and executed according to the input and output types of the adjacent operators.

[0167] In some embodiments, the input and output types of each operator can be stored in a database table. When the operator stream of the deep learning model is obtained, the database table is retrieved according to the identifier of each operator to obtain the input and output type of each operator, and then it is determined whether to insert quantization and / or dequantization operators between adjacent operators based on the input and output types of adjacent operators.

[0168] In some embodiments, a state machine is constructed, wherein the state machine includes multiple state identifiers representing multiple data types and multiple directed connection edges connecting every two of the state identifiers and representing the input type and output type of each operator, and then the operator flow is compared with the multiple directed connection edges of the state machine to determine whether the directed connection edges of the deep learning model are triggered. When a directed connection edge is triggered, if the source state identifier of the triggered directed connection edge does not match the input type of the operator represented by the triggered directed connection, the quantization operator and / or the dequantization operator is inserted before the corresponding operator of the deep learning model, or, if the target state identifier of the triggered directed connection edge does not match the output type of the operator represented by the triggered directed connection edge, the quantization operator and / or the dequantization operator is inserted after the corresponding operator of the deep learning model.

[0169] Step S140: deploy the operator flow to the designated acceleration unit for execution.

[0170] The operator finally obtained is an operator stream into which the inserted quantization and / or dequantization operator is inserted and which can be executed on the specified acceleration unit, and is deployed to the specified acceleration unit for execution.

[0171] In some embodiments, when the output type of a preceding operator of adjacent operators is different from the input type of a succeeding operator, it is determined to insert a quantization operator and / or a dequantization operator between the adjacent operators.

[0172] In some embodiments, the output type and the output type are classified into various precision types. When the output type of the previous operator is a high-precision type and the input type of the next instruction is a low-precision type, it is determined to insert a quantization operator; when the output type of the previous operator is a low-precision type and the input type of the next instruction is a high-precision type, it is determined to insert a dequantization operator.

[0173] In some embodiments, the output type of the current operator is a first data type, and the input type of the next operator is a second data type. If the first data type and the second data type belong to the same precision type, it is determined whether to insert a quantization operator or a dequantization operator based on the bits occupied by the first data type and the second data type in the memory; if the first data type and the second data type do not belong to the same precision type, it is determined whether to insert a quantization operator or a dequantization operator according to the precision type.

[0174] As an optional implementation method, a quantization instruction is inserted at the beginning of the operator stream to obtain quantized input data, and a dequantization instruction is inserted after the last executable instruction of the operator stream to obtain dequantized output data. The advantage of this is that the input data comes from the scheduling unit 220 or the memory 210, and the output data will be returned to the scheduling unit 220 or the memory 210. Through quantization and dequantization operations, it is ensured that the acceleration unit 230 obtains quantized input data, thereby reducing the data throughput of the acceleration unit 230, and the scheduling unit 220 or the memory 210 obtains dequantized output data, thereby ensuring the consistency of data on different processing units. In this way, the scheduling unit and the acceleration unit can collaboratively execute the deep learning model.

[0175] The compilation method of the deep learning model of the embodiment of the present disclosure determines whether to insert and execute quantization operators and / or dequantization operators between adjacent operators according to the input type and output type of the adjacent operators, so as to avoid inserting redundant quantization operators and / or dequantization operators, thereby ensuring the instruction simplification of the operator stream obtained thereby, and thereby improving the model efficiency and response speed of the deep learning model.

[0176] It should be noted that although the above descriptions are all about the improved tuning solutions for deep learning models at the algorithm level, in fact, the core ideas of the present disclosure can also be applied to firmware and hardware, for example, Figure 4 In the acceleration unit 230.

[0177] The commercial value of the disclosed embodiments

[0178] The disclosed embodiment utilizes the combination of hardware characteristics at the algorithm level to perform quantization operations, so that the reasoning performance of the deep learning model can be improved. For example, in resnet50, compared with the traditional tuning scheme, this case has a performance improvement of nearly 20%. The deep learning model currently has a wide range of successful application scenarios, making any minor improvements to the deep learning model so important, and this importance is not only reflected in the technical level, but also in the commercial level. Taking the field of face recognition as an example, video surveillance is collected through cameras, face images are recognized through neural network models, and face images are compared with faces stored in the cloud to identify criminals in surveillance videos. In the field of speech recognition, speech recognition is performed through neural network models to achieve simultaneous interpretation. These application scenarios can bring huge commercial benefits.

[0179] Those skilled in the art will appreciate that the present disclosure may be implemented as systems, methods, and computer program products. Therefore, the present disclosure may be specifically implemented in the following forms, namely complete hardware, complete software (including firmware, resident software, microcode), and may also be implemented in the form of a combination of software and hardware. In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0180] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by a processing unit, a device or a device, or used in combination with the program.

[0181] Computer readable signal media may include data signals propagated in baseband or as part of a chopped wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction system, apparatus, or device.

[0182] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0183] The computer program code for executing the disclosed embodiments may be written in one or more programming languages ​​or combinations. The programming languages ​​include object-oriented programming languages, such as JAVA, C++, and may also include conventional procedural programming languages, such as C. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0184] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A computing device comprising a memory, a scheduling unit, a designated acceleration unit, and a processor, wherein the memory stores instructions, and the processor reads the instructions to execute: In an operator stream of a deep learning model, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to input types and output types of the adjacent operators, the input types and the output types are classified as high-precision types and low-precision types, or the input types and the output types are data types, and the data types have corresponding precision types; Deploy the operator flow to the designated acceleration unit for execution; The inserting of a quantization operator and / or a dequantization operator between adjacent operators according to the input type and the output type of the adjacent operators comprises: When the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, a quantization operator and / or a dequantization operator is inserted between the adjacent operators.

2. The computing device according to claim 1, wherein when the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators comprises: When the output type of the preceding operator of the adjacent operator is a high-precision type and the input type of the succeeding operator is a low-precision type, a quantization operator is inserted; When the output type of the previous operator is a low-precision type and the input type of the subsequent operator is a high-precision type, a dequantization operator is inserted.

3. The computing device according to claim 1, wherein: When the output type of the preceding operator of the adjacent operator is different from the input type of the succeeding operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators includes: When the output type of the preceding operator of the adjacent operator is the first data type and the input type of the succeeding operator is the second data type, if the first data type and the second data type belong to the same precision type, inserting an operator or a dequantization operator based on the bits occupied by the first data type and the second data type in the memory; If the first data type and the second data type do not belong to the same precision type, a quantization operator or a dequantization operator is inserted according to the precision type.

4. The computing device according to claim 1, wherein: Inserting a quantization operator and / or a dequantization operator between the adjacent operators is also related to the input type and output type of the quantization operator and the dequantization operator.

5. The computing device according to claim 1, wherein: Using a predefined state machine, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to the input type and output type of the adjacent operators. The state machine includes multiple state identifiers representing multiple data types and multiple directed connection edges connecting every two of the state identifiers and representing the input type and output type of each operator.

6. The computing device according to claim 5, wherein: The inserting of a quantization operator and / or a dequantization operator between the adjacent operators according to the input type and the output type of the adjacent operators by using a predefined state machine comprises: Comparing each operator of the operator flow with a plurality of directed connection edges of the state machine to determine whether the directed connection edges are triggered; When a directed connection edge is triggered, if the source state identifier of the triggered directed connection edge does not match the input type of the operator represented by the triggered directed connection, the quantization operator and / or the dequantization operator is inserted before the corresponding operator, or, if the target state identifier of the triggered directed connection edge does not match the output type of the operator represented by the triggered directed connection edge, the quantization operator and / or the dequantization operator is inserted after the corresponding operator.

7. The computing device according to claim 5, wherein: In the state machine, the data type represented by the initial state identifier is consistent with the input type of the first operator of the operator stream.

8. The computing device according to claim 5, wherein: In the state machine, the input type of the operator represented by the directed connection edge is the same as the source state of the directed connection edge, and / or the output type of the operator represented by the directed connection edge is the same as the target state of the directed connection edge.

9. The computing device of claim 1, wherein: Also includes: The quantization operator is inserted at the beginning position of the operator stream, and the inverse quantization operator is inserted at the end position of the operator stream.

10. The computing device according to any one of claims 1 to 9, wherein: The input type and output type are respectively one of the following types: 16-bit floating point number, 24-bit floating point number, 32-bit floating point number, 64-bit floating point number, 32-bit signed integer, 16-bit signed integer, 8-bit signed integer, 8-bit unsigned integer.

11. A method for tuning a deep learning model, comprising: In an operator stream of a deep learning model, a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation is inserted between adjacent operators according to input types and output types of the adjacent operators, the input types and the output types are classified as high-precision types and low-precision types, or the input types and the output types are data types, and the data types have corresponding precision types; Deploy the operator flow to a designated acceleration unit for execution; The inserting of a quantization operator and / or a dequantization operator between adjacent operators according to the input type and the output type of the adjacent operators comprises: When the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, it is determined to insert a quantization operator and / or a dequantization operator between the adjacent operators.

12. The tuning method according to claim 11, wherein when the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators comprises: When the output type of the preceding operator of the adjacent operator is a high-precision type and the input type of the succeeding operator is a low-precision type, a quantization operator is inserted; When the output type of the previous operator is a low-precision type and the input type of the subsequent operator is a high-precision type, a dequantization operator is inserted.

13. The tuning method according to claim 11, wherein: When the output type of the preceding operator of the adjacent operator is different from the input type of the succeeding operator, inserting a quantization operator and / or a dequantization operator between the adjacent operators includes: When the output type of the preceding operator of the adjacent operator is the first data type and the input type of the succeeding operator is the second data type, if the first data type and the second data type belong to the same precision type, inserting an operator or a dequantization operator based on the bits occupied by the first data type and the second data type in the memory; If the first data type and the second data type do not belong to the same precision type, a quantization operator or a dequantization operator is inserted according to the precision type.

14. The tuning method according to claim 11, wherein: Inserting a quantization operator and / or a dequantization operator between the adjacent operators is also related to the input type and output type of the quantization operator and the dequantization operator.

15. The tuning method according to claim 11, wherein: Using a predefined state machine, the quantization operator and / or the dequantization operator is inserted between the adjacent operators according to the input type and output type of the adjacent operators. The state machine includes multiple state identifiers representing multiple data types and multiple directed connection edges connecting every two of the state identifiers and representing the input type and output type of each operator.

16. The tuning method according to claim 15, wherein: The inserting the quantization operator and / or the dequantization operator between the adjacent operators according to the input type and output type of the adjacent operators by using a predefined state machine comprises: Comparing each operator of the operator flow with a plurality of directed connection edges of the state machine to determine whether the directed connection edges are triggered; When a directed connection edge is triggered, if the source state identifier of the triggered directed connection edge does not match the input type of the operator represented by the triggered directed connection, the quantization operator and / or the dequantization operator is inserted before the corresponding operator, or, if the target state identifier of the triggered directed connection edge does not match the output type of the operator represented by the triggered directed connection edge, the quantization operator and / or the dequantization operator is inserted after the corresponding operator.

17. The tuning method according to claim 15, wherein: In the state machine, the data type represented by the initial state identifier is consistent with the input type of the first operator of the operator stream.

18. The tuning method according to claim 15, wherein: In the state machine, the input type of the operator represented by the directed connection edge is the same as the source state identifier of the directed connection edge, and / or the output type of the operator represented by the directed connection edge is the same as the target state identifier of the directed connection edge.

19. The tuning method according to claim 11, wherein: Also includes: The quantization operator is inserted at the beginning position of the operator stream, and the inverse quantization operator is inserted at the end position of the operator stream.

20. The tuning method according to any one of claims 11 to 19, wherein: The input type and output type are respectively one of the following types: 16-bit floating point number, 24-bit floating point number, 32-bit floating point number, 64-bit floating point number, 32-bit signed integer, 16-bit signed integer, 8-bit signed integer, 8-bit unsigned integer.

21. A method for compiling a deep learning model, the method being implemented in a compiler, comprising: Compile source code of the deep learning model into a first instruction sequence; Obtaining an operator stream executable on a designated acceleration unit from the first instruction sequence, in the operator stream, inserting a quantization operator for performing a quantization operation and / or a dequantization operator for performing a dequantization operation between adjacent operators according to input types and output types of adjacent operators, wherein the input type and the output type are classified as a high-precision type and a low-precision type, or the input type and the output type are data types, and the data types have corresponding precision types; Deploy the operator flow to a designated acceleration unit for execution; The inserting of a quantization operator and / or a dequantization operator between adjacent operators according to the input type and the output type of the adjacent operators comprises: When the output type of the preceding operator of the adjacent operators is different from the input type of the succeeding operator, a quantization operator and / or a dequantization operator is inserted between the adjacent operators.

22. A data center, comprising the computing device according to any one of claims 1 to 10.

Images