Processing unit, related device and method

Abstract

The present disclosure proposes a processing unit, related devices and methods. The processing unit includes: an instruction fetch unit for retrieving computer instructions; an instruction decoding unit for decoding the retrieved computer instructions; an instruction execution unit for executing the decoded computer instructions to achieve: converting a deep learning model in a specific format into a universal intermediate expression; inserting a maximum and minimum value collection node into the universal intermediate expression; converting the universal intermediate expression with the maximum and minimum value collection node inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression with the quantization node and dequantization node inserted back to the specific format to obtain a quantized model. The present disclosure improves the versatility of quantization of the acceleration unit.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more particularly, to a processing unit, related devices, and methods. Background Art

[0002] In the field of deep learning, inference refers to the application of a pre-trained deep learning model to real-world business scenarios. Regarding inference performance, in addition to hardware-level optimization, model quantization is an important approach to improving it at the algorithmic level. Model quantization involves converting the weight and input data for each node in a deep learning model from a high-precision quantization level to a low-precision quantization level, for example, converting 32-bit single-precision floating-point numbers (FP32) to 8-bit integer data (INT8). While this reduces accuracy, it helps reduce data throughput and storage requirements.

[0003] Deep learning models like TensorFlow and MXNet natively support quantization, but this quantization process is designed for CPUs and GPUs. For example, TensorFlow only supports quantization for a limited number of operations, such as convolution and matrix multiplication. However, for accelerators specifically designed for deep learning computations, not only these operations require quantization, but also many other operations. Therefore, existing quantization methods cannot meet the needs of accelerators specifically designed for deep learning computations.

[0004] Deep learning models are available in different formats, such as TensorFlow and MXNet. Existing technologies design quantization methods for each model, specifically for acceleration units that handle deep learning calculations. However, there is a lack of a universal quantization method that is suitable for all models. Summary of the Invention

[0005] In view of this, the present disclosure aims to propose a general quantization method and related device for an acceleration unit specifically processing deep learning calculations that is suitable for various deep learning models.

[0006] According to one aspect of the present disclosure, a processing unit is provided, comprising: an instruction fetch unit for retrieving computer instructions from a memory outside the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; an instruction execution unit for executing the decoded computer instructions to implement: converting a deep learning model in a specific format into a universal intermediate expression; inserting a maximum and minimum value collection node into the universal intermediate expression for collecting the maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the maximum and minimum value collection nodes have been inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the quantization node and dequantization node have been inserted back to the specific format to obtain a quantized model.

[0007] Optionally, inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized includes: determining a quantization factor based on the obtained maximum and minimum values ​​of the input to be quantized; and generating a quantization node and a dequantization node based on the quantization factor and inserting them into the universal intermediate expression.

[0008] Optionally, the quantization factor is configured in response to a user configuration request.

[0009] Optionally, converting the universal intermediate expression with the maximum and minimum value collection node inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized includes: converting the universal intermediate expression with the maximum and minimum value collection node inserted back to the deep learning model in the specific format; receiving a calibration data set so as to use an inference script to apply the calibration data set to the deep learning model converted back to the specific format, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input to be quantized.

[0010] Optionally, the calibration data set is specified in response to a user-specified instruction.

[0011] Optionally, inserting the maximum and minimum value collection node in the universal intermediate expression includes: determining an operation node in the universal intermediate expression; and inserting the maximum and minimum value collection node before the operation node selected by the user in the determined operation nodes.

[0012] Optionally, the instruction execution unit is also used to execute the decoded computer instructions to achieve: converting the general intermediate expression with the quantization node and the dequantization node inserted back to a specific format, obtaining the quantized model, and then converting the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit, and sending it to the acceleration unit.

[0013] Optionally, the instruction execution unit is also used to execute the decoded computer instructions to achieve: after sending the acceleration unit model to the acceleration unit, receiving the quantization accuracy obtained after the acceleration unit runs the acceleration unit model; determining whether the quantization accuracy meets the predetermined quantization accuracy standard; if not, redetermining the quantization strategy.

[0014] Optionally, in the deep learning model in the specific format, the operation of the maximum and minimum value collection node is defined and implemented in C++ language.

[0015] According to one aspect of the present disclosure, a computing device is provided, comprising: the processing unit as described above; a memory for storing the computer instructions; and an acceleration unit for running a deployed deep learning model.

[0016] According to one aspect of the present disclosure, a system on chip is provided, comprising an acceleration unit for running a deep learning model, wherein the deep learning model is quantized by the processing unit as described above and then deployed on the acceleration unit.

[0017] According to one aspect of the present disclosure, a data center is provided, comprising the computing device described above.

[0018] According to one aspect of the present disclosure, a deep learning model quantization method is provided, comprising: converting a deep learning model in a specific format into a universal intermediate expression; inserting a maximum and minimum value collection node into the universal intermediate expression to collect the maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the maximum and minimum value collection nodes have been inserted back into the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; converting the universal intermediate expression into which the quantization and dequantization nodes have been inserted back into the specific format to obtain a quantized model.

[0019] Optionally, inserting quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized includes: determining a quantization factor based on the obtained maximum and minimum values ​​of the input to be quantized; and generating a quantization node and a dequantization node based on the quantization factor and inserting them into the universal intermediate expression.

[0020] Optionally, converting the universal intermediate expression with the maximum and minimum value collection node inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized includes: converting the universal intermediate expression with the maximum and minimum value collection node inserted back to the deep learning model in the specific format; receiving a calibration data set so as to use an inference script to apply the calibration data set to the deep learning model converted back to the specific format, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input to be quantized.

[0021] Optionally, inserting the maximum and minimum value collection node in the universal intermediate expression includes: determining an operation node in the universal intermediate expression; and inserting the maximum and minimum value collection node before the operation node selected by the user in the determined operation nodes.

[0022] Optionally, after converting the universal intermediate expression with the quantization node and the dequantization node inserted back to a specific format to obtain the quantized model, the method further includes: converting the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit, and sending it to the acceleration unit.

[0023] Optionally, after sending the acceleration unit model to the acceleration unit, the method further includes: receiving the quantization accuracy obtained after the acceleration unit runs the acceleration unit model; determining whether the quantization accuracy meets a predetermined quantization accuracy standard; if not, redetermining the quantization strategy.

[0024] In the embodiment of the present disclosure, regardless of the format of the deep learning model, it will be converted into a universal intermediate expression, and then calibration and quantization will be performed in this universal intermediate expression. Specifically, during calibration, a maximum and minimum value collection node is inserted into the universal intermediate expression to collect the maximum and minimum values ​​of the input to be quantized, and then the universal intermediate expression with the maximum and minimum value collection node inserted is converted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized, thereby achieving that the calibration process is independent of the type of deep learning model. During quantization, quantization and dequantization nodes are inserted into the universal intermediate expression according to the maximum and minimum values ​​of the input to be quantized, and then the universal intermediate expression with the quantization node and the dequantization node inserted is converted back to the specific format to obtain the quantized model, thereby achieving that the quantization process is also independent of the type of deep learning model. Therefore, it proposes a universal quantization method for acceleration units that are suitable for various deep learning models and are specifically designed to handle deep learning calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0026] Figure 1is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0027] Figure 2 is an internal structure diagram of a server in a data center according to an embodiment of the present disclosure;

[0028] Figure 3 This is an internal structural diagram of a processing unit and an acceleration unit within a server according to an embodiment of the present disclosure;

[0029] Figure 4 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present disclosure;

[0030] Figure 5 This is an exemplary hierarchical architecture diagram for a deep learning model running platform;

[0031] Figure 6 The model states of the deep learning model at different stages provided by the embodiments of the present disclosure are shown;

[0032] Figure 7 A flowchart of a deep learning model quantization method according to an embodiment of the present disclosure is shown;

[0033] Figure 8A shows a flowchart of the execution of an operation node in a general deep learning model; for comparison, Figure 8B The flowchart of the operation node after quantization is shown;

[0034] Figure 9 shows a calibration flow chart according to one embodiment of the present disclosure;

[0035] Figure 10 A quantization flow chart according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0037] The following terms are used in this document.

[0038] Deep Learning Model: Deep learning is a new research direction in the field of machine learning (ML). It was introduced to bring ML closer to its original goal: artificial intelligence (AI). Deep learning learns the inherent patterns and representational hierarchies of sample data. The information gained during this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have human-like analytical learning capabilities and the ability to recognize data such as text, images, and sound. The model used in deep learning is called a deep learning model. Deep learning models have different formats depending on the model framework they rely on, and can be categorized into different formats, such as TensorFlow, PyTorch, and MXNet.

[0039] Acceleration unit: A processing unit designed to improve the data processing speed in some special-purpose fields (for example, image processing, various operations of deep learning models, etc.) where traditional processing units are inefficient. In the embodiments disclosed herein, the processing unit is a special processing unit designed mainly to accelerate the operation processing speed of deep learning models.

[0040] Processing unit: A unit within a data center server that performs traditional processing (not for complex computations like image processing and deep learning models). In addition to performing these non-image processing and deep learning model-related complex computations, the processing unit also schedules the acceleration units, assigning them tasks. Processing units can take various forms, including CPUs, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0041] Quantization: The act of converting the inputs of operation nodes in a deep learning model, as well as the weight parameters and other parameters of the operation nodes, from high-precision data types to low-precision data types, thereby reducing the requirements for data throughput and storage space.

[0042] The following briefly introduces the quantization process with a specific example of quantization.

[0043] During quantization, in order to convert the model parameters and input parameters of the node into low-precision data types, we first need to know the value range of the model parameters and input parameters. For example, the maximum value is 100 and the minimum value is -100. If converted to binary 8-bit integers, let 2 8 -1, that is, 11111111 corresponds to the maximum value 100, and 0, that is, 00000000 corresponds to the minimum value -100. At this time, it is equivalent to dividing the length of the interval between -100 and 100, a total of 200, into (28 -1) small intervals, the length of each small interval is 200 / (2 8 -1) = 0.784. Thus, for any value between -100 and 100, its cell interval can be determined. The left or right endpoint of the cell interval is used as the quantized value. For example, for 50, [50 - (-100)] / 0.784 = 191.3, which corresponds to the 191st cell interval, with the left endpoint being 191, or 10111111, and the quantized value being 101111111.

[0044] In the case of linear quantization, quantization can be performed using the following formula. δ is the scaling factor, which is equivalent to the difference between the quantized value and the value before quantization. δ is the maximum value x of the input to be quantized. max , minimum value x min It will be determined by min , x max 】 is quantized to any integer between 0 and n-1 (including 0 and n-1). Let q(x) be the value of the quantized input x, and the formula is:

[0045]

[0046] Among them, clamp is the limit function, and formula 1 means round((xx min ) / δ) is limited to 0 to n-1. If round((xx min ) / δ) is greater than n-1, then the value of formula 1 is n-1; if round((xx min ) / δ) is less than 0, the value of formula 1 is 0; if round((xx min ) / δ) is between 0 and n-1, then the value of Formula 1 is round((xx min ) / δ).

[0047] The scaling factor δ is calculated as follows:

[0048] δ=[max(x max )-min(x min )] / (n-1) Formula 2

[0049] Taking into account the zero point problem (error-free representation of floating point numbers as 0), it is necessary to add an offset to the quantization method, namely z q δ and z q is the maximum value x of the input to be quantized max , minimum value x min decided.

[0050] The scaling factor δ after taking into account the zero point problem is calculated as follows:

[0051]

[0052] Due to the zero point problem, we cannot simply determine which of 0 to n-1 it is quantized based on the value of (x / δ). We must consider the offset z q The offset z q Calculated as follows:

[0053]

[0054] Considering the offset z q Then, the quantized value q(x) is determined according to the pre-quantized value x according to the following formula:

[0055]

[0056] Dequantization: The reverse process of quantization, converting the inputs, weights, and other parameters of operation nodes in a deep learning model from low-precision data types to high-precision data types. The dequantization process is the reverse of the quantization process described above, so it will not be described in detail.

[0057] Operation node: The smallest unit of independent operation in a deep learning model, which receives input and generates output after operating on its own weight parameters or parameters in other models (such as hyperparameters). A deep learning model can include various specific operations such as convolution and pooling, and has various operation nodes including convolution operation nodes and pooling operation nodes. There are multiple layers in a deep learning model, and each layer has multiple operation nodes. The output of each operation node is the input of the operation node of the next layer. Specifically, the operation node includes the program and related data for the specific operation. For example, the convolution operation node includes the program code used for the convolution operation and some data used in the convolution.

[0058] Common Intermediate Representation: Deep learning models have different formats, depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet. The code representations of these deep learning models also differ. This poses significant challenges to the universality of deep learning model quantification. Common Intermediate Representation (CIR) converts the various deep learning model code representations into a common format. It analyzes the meaning of each code statement in a deep learning model and translates it into a common representation based on its meaning. This ensures that code statements with the same meaning across different deep learning models are expressed identically in the CIR. Currently, there are tools available that convert the representations of different deep learning models into a CIR.

[0059] Calibration: As in the quantization example above, quantization first requires determining the value range of the inputs of the operation nodes in the deep learning model. Only by determining the maximum and minimum values ​​of the input to be quantized can the maximum and minimum values ​​of the input to be quantized be mapped to the maximum and minimum values ​​after quantization, thus completing the quantization of the input to be quantized. Calibration is the process of determining the maximum and minimum values ​​of the input to be quantized.

[0060] A Maximum / Minimum Value Collection node is a program and data program inserted at a specific location in a program to collect the maximum and minimum values ​​generated at that location during program execution. It includes the program code to collect the values ​​generated at that location and determine the maximum and minimum values, as well as the data required for this process. This location is typically before an operation node. By inserting this node before an operation node of interest, you can collect the various input values ​​fed into that operation node and determine the maximum and minimum values.

[0061] Quantization and Dequantization nodes are placed at specific locations in the program to perform the aforementioned quantization and dequantization operations, respectively, and their associated data. This way, when a deep learning model runs on a quantization node, quantization is performed; when it runs on a dequantization node, dequantization is performed.

[0062] Quantized model: The model generated by the above deep learning model after quantization.

[0063] Quantization strategy: The quantization method used when quantizing a deep learning model. Quantization methods can include linear quantization and other methods. The quantization strategy refers to the selection of these quantization methods.

[0064] Quantization factors: When a quantization method is selected, it usually has some factors that are used in the quantization process. These factors are called quantization factors. For example, in the case of linear quantization, the scaling factor and offset are quantization factors.

[0065] Calibration dataset: A dataset used during the calibration process for a deep learning model. Calibration is completed by inputting data from this dataset into the deep learning model and detecting the maximum and minimum values ​​of the input at a specific location within the deep learning model.

[0066] Inference script: A code snippet and related data used to run a deep learning model for actual inference.

[0067] Quantified accuracy: A measure of the accuracy of inference after actual use of a deep learning model.

[0068] Application environment of the present disclosure

[0069] The embodiments of the present disclosure propose a universal quantization scheme suitable for various deep learning models. The entire compilation optimization scheme is relatively universal. The model after quantization is completed on the processing unit can be used for various hardware devices that execute the model, for example, for data centers, for AI (artificial intelligence) acceleration units, for GPUs (graphics processing units), for IOT (Internet of Things) devices that can execute deep learning models, embedded devices, etc. This compilation optimization method is independent of the hardware on which the model is ultimately deployed. However, for the sake of exemplary description, the following description will mainly focus on the data center as the application scenario. Those skilled in the art should understand that the embodiments of the present disclosure can also be applied to other application scenarios.

[0070] Data Center

[0071] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's network infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Deep learning, a key AI technology, has been widely applied to big data analytics in data centers.

[0072] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in Figure 1, this is the hierarchical inter-networking model. This model consists of the following parts:

[0073] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0074] Access switch 130: Access switches 130 connect servers 140 to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, and are therefore also called top-of-rack switches. They physically connect to the servers.

[0075] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0076] Core switch 110: Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for aggregation switch 120. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0077] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0078] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with other aggregation switches 120 being used only in the event of a failure (dashed lines in the figure above). This means that horizontal scalability is impossible at the aggregation switch 120 level because even if multiple aggregation switches 120 are added, only one is still operational.

[0079] server

[0080] Since the server 140 is the real processing device in the data center, Figure 2 The figure shows a block diagram of the internal structure of a server 140. The server 140 includes a memory 210, a processing unit cluster 270 and an acceleration unit cluster 280 connected by a bus. The processing unit cluster 270 includes a plurality of processing units 220. The acceleration unit cluster 280 includes a plurality of acceleration units 230. In the embodiment of the present disclosure, the acceleration unit is a special processing unit designed mainly to accelerate the operation processing speed of the deep learning model. It can be embodied as a processing unit specially designed for deep learning model operation processing, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA). The processing unit is a processing unit that schedules the acceleration unit and allocates a sequence of instructions to be executed to each acceleration unit. It can take the form of a processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) and the like.

[0081] The architecture design of the traditional processing unit makes the control unit and storage unit occupy a large part of the space in the architecture, while the space occupied by the computing unit is insufficient. Therefore, it is very effective in terms of logical control, but not efficient enough in terms of large-scale parallel computing. Therefore, various special acceleration units have been developed to more effectively improve the computing speed of calculations for different functions and different fields. The acceleration unit proposed in the present invention is a processing unit dedicated to accelerating the computing speed of deep learning models. It is a processing unit that uses a data-driven parallel computing architecture to process a large number of operations (such as convolution, pooling, etc.) of each deep learning model node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each deep learning model node are closely related in the entire computing process and will be used frequently, using the existing processing unit architecture, since the memory capacity in the core of the processing unit is very small, a large number of external core memories must be frequently accessed, resulting in inefficient processing. By using this acceleration unit specifically designed to accelerate the computing processing speed of deep learning models, each core of the unit has on-chip memory with a storage capacity suitable for deep learning model calculations, thereby avoiding frequent access to memory outside the core, it can greatly improve processing efficiency and computing performance.

[0082] The acceleration unit 230 must accept the scheduling of the processing unit 220. Figure 2 As shown, the memory 210 stores various deep learning models, including the nodes of these models and the weight data of the nodes. These deep learning models are used when needed. Figure 2 A processing unit 220 is deployed to an acceleration unit 230. That is, the processing unit 220 can send the address of the parameters in the model (such as the weights of each node) in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually uses the deep learning model for calculation, it will directly address these parameters in the memory 210 according to the addresses of these parameters (such as weights) in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the deep learning model for calculation, the processing unit 220 will also send the input of the model to the acceleration unit 230 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these inputs and the parameters in the model (such as weights). The embodiment of the present disclosure mainly focuses on the quantization process before the processing unit 220 deploys the deep learning model to the acceleration unit 230. Quantization will be described in detail later.

[0083] Internal structure of the processing unit and acceleration unit

[0084] The following combination Figure 3The internal structure diagram of the processing unit 220 and the acceleration unit 230 specifically illustrates how the processing unit 220 schedules the acceleration unit 230 to work.

[0085] like Figure 3 As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0086] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 3 The register file 229 shown is used to store a register for instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0087] After fetching the instruction, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0088] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0089] After the instruction issuing unit 225 sends the instruction to the instruction executing unit 226, the instruction executing unit 226 begins executing the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is an instruction for deep learning model inference, the instruction executing unit 226 will no longer execute the instruction. Instead, it will send the instruction to the acceleration unit 230 via the bus for execution.

[0090] The acceleration unit 30 includes multiple cores 236 ( Figure 3 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0091] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0092] Direct Memory Access (DMA) mechanism 235 is a feature provided by some computer bus architectures that enables data to be written directly from an attached device to the computer's mainboard memory. This greatly improves data access efficiency compared to a method where all data transfers between devices must pass through the processing unit. Because of this mechanism, the cores of the acceleration unit 230 can directly access memory 210 and read parameters (such as the weights of each node) in the deep learning model, greatly improving data access efficiency.

[0093] The command processor 237 assigns instructions sent from the processing unit 220 to the acceleration unit 230 to be executed by the core 236. The instruction execution unit 226 sends the acceleration unit 230 a sequence of pending instructions to be executed. After entering the bus channel 231, the pending instruction sequence is cached in the command processor 237, which then selects a core 236 and assigns the instruction sequence to it for execution. The command processor 237 also manages synchronization between the cores 236.

[0094] Acceleration unit core

[0095] Figure 4 FIG. 4 is a diagram of the internal structure of an acceleration unit core according to an embodiment of the present disclosure.

[0096] In one embodiment, Figure 4 As shown, the acceleration unit core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0097] The instruction sequence assigned by the command processor 237 to the acceleration unit core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and assigns them to the tensor engine 310 or the pooling engine 320 for execution based on the nature of the instruction. The tensor engine 310 is responsible for processing operations such as convolution and matrix multiplication in deep learning models. The pooling engine 320 is responsible for processing pooling operations in deep learning models. The memory copy engine 330 is a unit specifically responsible for data copying. This data copying involves copying some data from the on-chip memory 360 to the shared memory of each core 236, or to the on-chip memory 360 of other cores 236, due to potential overflow of the on-chip memory 360. The sequencer 350 determines whether to assign the instruction to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 based on the nature of the operation, such as convolution, matrix multiplication, pooling, or data copy.

[0098] The on-chip memory 360 is an in-core memory that stores the weight parameters in the deep learning model, as well as the input parameters and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning model (for example, hyperparameters in the neural network model) in addition to the weight parameters. As described above, in the process of the processing unit 220 pre-configuring the deep learning model in the acceleration unit 230, the processing unit 220 sends the address of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the acceleration unit 230 takes it out from the corresponding position of the memory 210 when the actual deep learning model is operated and places it in the on-chip memory 360. For other parameters, the acceleration unit 230 takes it out from the corresponding position of the memory 210 when the actual deep learning model is operated and places it in the constant buffer 370. Furthermore, when the instruction to actually start inference is assigned by the command processor 237 to the core 236 for execution, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. Furthermore, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0099] Quantitative Operations

[0100] Below, combined with the attached Figure 5 , describing an exemplary layered architecture diagram for a deep learning model execution platform. Referring to the figure, from top to bottom, the architecture diagram includes an application layer 401, a framework layer 402, and a function layer 403.

[0101] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, and recommendation 407. These applications are built using this architecture and can also call the architecture to provide runtime interfaces to gain reasoning capabilities within the application.

[0102] The framework layer 402 integrates open-source platforms such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to enable continued optimization and improvement of various algorithms. TensorFlow 408 is a symbolic mathematical system based on data flow programming and is widely used to implement various machine learning algorithms. MXNet 409 is the deep learning library of choice for Amazon. Caffe 410, or Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modularity.

[0103] Functional layer 403 includes a compilation stack 403 and a runtime stack 404. Compiler stack 403 is used for conversion (converter 411), quantization (quantization) 412, optimization (optimization) 413, and compilation (compile) 414 of various models. Conversion 411 converts the model's internal data into a common intermediate representation (IR) format. Quantization 412 converts parameters such as weights in deep learning models and inputs to deep learning models from high-precision data types to low-precision data types. Optimization 413 fuses operators within the model and performs operations such as multi-model optimization linking. Compilation 414 optimizes the model based on hardware (e.g., neural network processors) to generate a binary model that the hardware can recognize. Runtime stack 404 includes runtime API 415, execution manager 416, user-mode driver 417, and kernel-mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized runtime API 415 provides various runtime callable interfaces. The user-mode driver 417 provides kernel-mode hardware commands and resource scheduling. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0104] The quantization operation referred to herein is the quantization 412 mentioned above, which mainly includes converting parameters such as weights in the deep learning model and the input to the deep learning model from high-precision data types to low-precision data types. This is explained below using a convolutional neural network as an example. A convolutional neural network includes multiple layers (convolutional layers, pooling layers, etc.), each of which has multiple operation nodes. An operation node is the smallest computing unit in a deep learning model that performs operations based on the input to obtain the output passed to the next layer. The features output by the nodes of each convolutional layer are transmitted to the nodes of the next convolutional layer as the input received by the next convolutional layer. Each layer obtains a feature map by performing matrix multiplication and summation calculation using the convolution kernel. The convolution kernel is a two-dimensional matrix, and each value in the two-dimensional matrix can be regarded as a weight parameter. Since the convolutional neural network has multiple layers, there are multiple weight parameters in the convolution kernel of each layer, so the number of weight parameters is huge. Corresponding to the weight parameters, the number of input parameters in the input matrix multiplied by the convolution kernel is also huge. During the model training phase, weight parameters are generally stored and calculated using 32-bit floating-point numbers. The quantization operation is to convert the weight parameters from 32-bit floating-point numbers to a lower-precision data type for storage and calculation, such as converting them to 8-bit integers (signed integers or unsigned integers) or 16-bit floating-point types for storage and calculation. The quantization operation can only convert a portion of the weight parameters, for example, only converting the convolution kernels of a portion of the convolution layer, while the convolution kernels of the remaining convolution layers remain unchanged. Corresponding to the weight parameters, the quantization operation also converts each parameter in the actual input matrix to a lower-precision data type for storage and calculation. The processing unit 220 usually deploys the quantized deep learning model to the acceleration unit 230 in the form of instructions only after completing the quantization operation. That is, the processing unit 220 converts the weights of the deep learning model stored in the memory 210 from a high-precision data type to a low-precision data type, thereby improving data throughput and saving storage space. Then, the processing unit 220 sends the addresses of these quantized weights in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually runs the deep learning model for inference, it can directly retrieve the data from the memory 210 according to these addresses and put it into the on-chip core 360 ​​of the acceleration unit 230 for execution. At the same time, when the deep learning model is actually running, the processing unit 220 also converts the quantized model actual input from a high-precision data type to a low-precision data type, and notifies the acceleration unit 230 of their addresses in the memory 210 in the form of instructions, so that the acceleration unit 230 can use these inputs and weight parameters to run the model, thereby improving data throughput and saving storage space.

[0105] Figure 6 The model states of the deep learning model at different stages provided by the embodiment of the present disclosure are shown. Figure 6As shown, the original model 501 is a pre-trained high-precision deep learning model, and the quantized model 502 is a deep learning model obtained by quantizing the original model 501. The quantization operation can be performed in steps, for example, first calibrating to obtain a calibrated model, and then performing a quantization operation on the calibrated model to obtain the quantized model 502.

[0106] Calibration is the act of determining the value range of the input of the operation node in the deep learning model, thereby providing a basis for quantization. Only after the maximum and minimum values ​​of the input to be quantized are determined can the maximum and minimum values ​​of the input to be quantized be mapped to the maximum and minimum values ​​after quantization, thereby completing the quantization of the input to be quantized. Calibration can be achieved by inserting a maximum and minimum value collection node at a specific location in the deep learning model. After inserting the maximum and minimum value collection node, when the maximum and minimum value collection node is executed during the running of the deep learning model, the maximum and minimum value collection node can collect various input values ​​and obtain the maximum and minimum values. This specific location is generally in front of the operation node (such as the convolution node). By inserting this node in front of the operation node of interest, various input values ​​of the input to the operation node can be collected to obtain the maximum and minimum values.

[0107] The following combination Figure 8A -B, describe the quantization process in detail.

[0108] Figure 8A The diagram shows the operation process of the operation node of the deep learning model without quantization. That is, the operation node 702 (such as a convolution node) in the deep learning model receives an input 701 to the operation node 702 and generates an output operation result 703 (such as a convolution result).

[0109] like Figure 8B As shown, in the case of quantization, the normal operation node 702 is converted into a quantized version of the operation node 805. The operation node 702 mainly includes the program code and code-related data used for the operation. When converting the normal operation node 702 into the quantized version of the operation node 805, it is generally necessary to modify the parameter definitions in the code of the operation node 702 to the quantized parameter definitions (such as changing a 32-bit floating point number to an 8-bit integer number), and modify the code-related data, such as some constants in the code. Then, a quantization node 804 is added before the quantized version of the operation node 805 to quantize the input of the operation node and the parameters such as the weight in the operation node. After the quantized version of the operation node 805, a dequantization node 806 is added to dequantize the quantized input and weight parameters. During actual operation, a calibration process is first performed, that is, the maximum and minimum value collection node collects the maximum value 803 and minimum value 802 of the input to be quantized 801. Then, the quantization node 804 begins quantization.

[0110] After the quantization level is determined, the range of the quantized input can generally be determined. For example, if the input of a deep learning model is quantized to an 8-bit integer, the minimum value of the quantized input is 0 and the maximum value is 2. 8 -1 = 255, but to map the quantized input to an integer between 0 and 255, we also need to know the maximum and minimum values ​​of the input. Calibration is the process of determining the maximum and minimum values ​​of the input to be quantized, providing a basis for quantization. The detailed quantization formula was discussed in the previous introduction to quantization concepts, so I will not repeat it here.

[0111] The quantization process of the quantization node 804 is described above. Figure 8B As shown, a dequantization node 806 is inserted after the quantization version operation node 805. The dequantization process is the reverse process of the above quantization process, so it is not described in detail. After dequantization, the result output 807 is obtained.

[0112] like Figure 6 As shown, some deep learning models combine the calibration and quantization operations described above. This means that the original model 501 is input and the quantized model 502, which has undergone calibration and quantization, is output. After the quantization operation is completed, the high-precision model is converted to a low-precision model. The quantized model 502 is then converted to an acceleration unit model 503. Acceleration unit model 503 is still a low-precision model. However, acceleration unit model 503 can be recognized by the instruction set of acceleration unit 230 and can therefore be executed on acceleration unit 230, whereas the original model 501 and quantized model 502 can only be recognized by the instruction set of processing unit 220 and can therefore be executed on processing unit 220.

[0113] It should be pointed out that Figure 6 It is only used to illustrate the execution status of the above model, but cannot be used to illustrate the storage status of the above model. In fact, the original model 501, the quantized model 502 and the acceleration unit model 503 are all stored in the memory 210, and will only be loaded into the processing unit 220 or the acceleration unit 230 during execution.

[0114] It should also be pointed out that the model commonly referred to actually contains code and data samples that implement the corresponding algorithm. Code exists in many forms, such as source code (such as Java, Python implementation) and executable code (binary code). The source code can only be recognized and executed by the processor after it is compiled into executable code. Therefore, the above-mentioned conversion of the quantization model 502 into the acceleration unit model 503, so as to be recognized by the instruction set of the acceleration unit 230, refers to modifying part of the source code of the quantization model and compiling it into executable code, and then the executable code can be recognized by the instruction set of the acceleration unit 230, and so on. However, in order to simplify the description, such a strict description will not be performed in each step in this article.

[0115] The quantization method of the embodiment of the present disclosure

[0116] Deep learning models are available in various formats, such as TensorFlow and MXNet. Existing technologies design quantization methods for each format or model, specifically for accelerators that handle deep learning computations. However, a universal quantization method applicable to all formats is lacking. In the disclosed embodiments, regardless of the format of the deep learning model, it is converted into a universal intermediate representation, which is then calibrated and quantized within this universal intermediate representation. Therefore, a universal quantization method for accelerators that handle deep learning computations, applicable to various deep learning models, is proposed.

[0117] In the embodiment of the present disclosure, the processing unit 220 first converts the deep learning model in the specific format into a universal intermediate expression. The universal intermediate expression is a universal format that converts the code expressions of deep learning models in various formats. It analyzes the meaning of each code statement in the deep learning model and translates the statement into a universal expression according to the meaning of the code statement, so that the code statements with the same meaning in different deep learning models are expressed consistently in the universal intermediate expression. It is Figure 5 The conversion (converter) 411 function of the compilation stack 403 is completed, which can be achieved by existing means. Figure 9 As shown, original deep learning models 501 in various formats, including a deep learning model in format AC, are converted into a universal intermediate representation 504.

[0118] Then, if Figure 9 As shown, the processing unit 220 inserts a maximum and minimum value collection node for collecting the maximum and minimum values ​​of the input to be quantified into the universal intermediate expression 504, thereby forming a universal intermediate expression 505 into which the maximum and minimum value collection node is inserted. The maximum and minimum value collection node 505 is a program and related data inserted at a specific position in the deep learning model to collect the maximum and minimum values ​​of the values ​​generated at the specific position when the deep learning model is running. The specific position is generally in front of the operation node. By inserting this node in front of the operation node of interest, various input values ​​of the input to the operation node can be collected to obtain the maximum and minimum values. Therefore, the process may include determining the operation nodes in the universal intermediate expression, and inserting the maximum and minimum value collection node before the operation node selected by the user in the determined operation nodes.

[0119] When converting the deep learning model in the specific format into a universal intermediate representation, the meaning of each code statement in the deep learning model is parsed and translated into a universal representation based on the meaning of the code statement. Therefore, the operation nodes in the universal intermediate representation can be clearly identified based on the meaning of the code statement. For example, a code segment that completes a convolution operation is treated as a convolution node, and a code segment that completes a pooling operation is treated as a pooling node.

[0120] After determining each operational node, the user can display these nodes to the user, who can be an administrator debugging a deep learning model. After displaying these nodes, the user can select the node to be calibrated through the interface. A maximum and minimum value collection node can then be inserted before the selected operational node. This approach allows for calibration only of the operational nodes of interest, while unrelated operational nodes remain unaffected. This approach is equivalent to a non-invasive probe method, significantly expanding the scope of quantification.

[0121] Then, if Figure 9 As shown, processing unit 220 converts the universal intermediate representation 505, into which the maximum and minimum value collection nodes are inserted, back into the model 501 in the specific format for calibration, thereby obtaining the maximum and minimum values ​​of the input to be quantized. The reason for converting the universal intermediate representation back into the deep learning model in the specific format before calibration is that this allows maximum and minimum value information to be collected for the operation nodes of interest, while leaving unaffected operation nodes unaffected, thereby increasing quantization flexibility.

[0122] In this process, the universal intermediate expression with the maximum and minimum value collection node inserted can be first converted back to the deep learning model in the specific format, and then a calibration data set can be received so that the calibration data set can be applied to the converted deep learning model in the specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input to be quantified.

[0123] Converting the universal intermediate expression with the maximum and minimum value collection nodes inserted back to the deep learning model in the specific format is the reverse process of converting the deep learning model in the specific format into the universal intermediate expression. Figure 5 The conversion 411 function of the compilation stack 403 is completed.

[0124] The inference script is a script used to run the deep learning model and use it for actual reasoning. The calibration dataset is the dataset applied to the deep learning model during the calibration process, mainly including a variety of raw data samples that are generally input to the deep learning model. The various raw data samples that are commonly used are placed in the calibration dataset for input into the deep learning model during calibration to obtain the required maximum and minimum values. It is worth noting that this raw data sample is the input of the entire deep learning model, but the input for obtaining the maximum and minimum values ​​in the calibration refers to the specific position where the maximum and minimum value collection node is inserted in the universal intermediate expression. When the deep learning model is running, the raw data sample is input into the deep learning model. When the model runs to this specific position, the maximum and minimum value collection node collects the input of this specific position, and the maximum and minimum values ​​of the input of this specific position are obtained.

[0125] After running the inference script, the maximum and minimum value collection node automatically collects the maximum and minimum value information during the operation and writes it to the specified storage location in a predetermined format. During the quantization process, the collected maximum and minimum value information can be retrieved from the specified storage location and quantized with reference to this information. The universal intermediate expression, i.e., the calibration model, into which the maximum and minimum value collection node is inserted can run in the deep learning model framework only if the operation of the maximum and minimum value collection node is defined in the deep learning model of the specific format and implemented in C++.

[0126] In one embodiment, the calibration dataset is specified in response to a user-specified instruction. Candidate calibration datasets can be set, and a calibration dataset from the candidate calibration datasets can be specified using user-specified instructions. Alternatively, a user can create a custom calibration dataset and then specify it using user-specified instructions. Because the calibration dataset input during the calibration phase is user-specified, the user can adjust the calibration dataset at any time, greatly facilitating re-quantification and re-deployment in practical situations. This approach enhances quantification flexibility.

[0127] Then, if Figure 10 As shown, processing unit 220 inserts quantization and dequantization nodes into a universal intermediate representation 504 converted from the original deep learning model 501 in various formats, based on the obtained maximum and minimum values ​​of the input to be quantized, to form a universal intermediate representation 506 with quantization and dequantization nodes inserted. Quantization and dequantization nodes are programs and related data inserted at specific locations in the program, respectively used to perform the aforementioned quantization and dequantization operations. Thus, when the deep learning model reaches a quantization node, the quantization operation is performed; when it reaches a dequantization node, the dequantization operation is performed.

[0128] In one embodiment, a quantization factor may be determined based on the obtained maximum and minimum values ​​of the input to be quantized, and then a quantization node and a dequantization node may be generated based on the quantization factor and inserted into the universal intermediate expression.

[0129] Quantization factors are factors used in the quantization process. In other words, the quantization process is determined by these factors, which are determined by the maximum and minimum input values ​​obtained through calibration. The previous description introduced the scaling factor and offset in linear quantization, which are two quantization factors. These two factors primarily control the quantization process. However, in other quantization scenarios, the quantization factors may be other factors.

[0130] The quantization node and the dequantization node are essentially the programs and related data for performing the above-mentioned quantization and dequantization operations respectively. Once the quantization factor is determined, the related programs and data are actually determined. Therefore, the quantization node and the dequantization node can be generated according to the quantization factor. The process of inserting the quantization node and the dequantization node into the general intermediate expression is as follows: Figure 8B As shown, the process generally includes changing the operation node 702 to a quantized version operation node 805 , then inserting a quantization node 804 in front of the quantized version operation node 805 , and inserting a dequantization node 806 behind the quantized version operation node 805 .

[0131] In one embodiment, the quantization factor is configured in response to a user configuration request. As previously described, in the case of linear quantization, the quantization factor can be configured as a scaling factor and an offset, but other quantization methods can also be used, and in these other quantization methods, the quantization factor can be configured as other quantization factors. Therefore, the quantization method of the embodiment of the present disclosure can try a variety of different quantization configurations, adjust quantization according to actual conditions, and improve quantization accuracy.

[0132] Then, if Figure 10 As shown, the processing unit 220 converts the general intermediate expression 506 inserted with the quantization node and the dequantization node back to the deep learning model in a specific format to obtain the quantized model. Figure 5 The conversion 411 function of the compilation stack 403 is completed.

[0133] Then, the processing unit 220 can convert the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit and send it to the acceleration unit. Figure 6 It has been explained in detail, so I won’t repeat it here.

[0134] In addition, after the processing unit 220 sends the acceleration unit model to the acceleration unit 230, the acceleration unit 230 runs the acceleration unit model, obtains the quantization accuracy after running, and returns it to the processing unit 220. The processing unit 220 determines whether the quantization accuracy meets the predetermined quantization accuracy standard. If it does, there is no need to adjust the quantization strategy. If it does not meet, redefine the quantization strategy. The quantization strategy refers to the quantization method used when quantizing the deep learning model, such as a linear quantization method or a nonlinear quantization method. Therefore, if the quantization accuracy does not meet the predetermined quantization accuracy standard, the quantization method can be adjusted, for example, the linear quantization method can be changed to a nonlinear quantization method. For example, the quantization accuracy returned by the acceleration unit 230 is 0.92, and the predetermined quantization accuracy is greater than or equal to 0.95. At this time, it is considered that the quantization accuracy of 0.92 obtained during actual execution does not meet the predetermined quantization accuracy standard, and the quantization strategy needs to be redetermined, for example, the quantization method is adjusted to another known quantization method.

[0135] As can be seen from the above, the quantization method of the disclosed embodiment mainly modifies the deep learning model and does not restrict user input. In actual use, user input can vary. Therefore, the disclosed embodiment can be applied to deep learning models for various applications, such as single-input image classification and detection, or multi-input GAN generation network.

[0136] In addition, the embodiment of the present disclosure collects data information by inserting maximum and minimum collection nodes, and each format of the deep learning model provides an interface for inserting nodes, so it has extremely strong versatility.

[0137] like Figure 7 As shown, the embodiment of the present disclosure also provides a deep learning model quantization method, including:

[0138] Step 610: Convert the deep learning model in a specific format into a universal intermediate representation;

[0139] Step 620: insert a maximum and minimum value collection node into the general intermediate expression to collect the maximum and minimum values ​​of the input to be quantized;

[0140] Step 630: Convert the universal intermediate expression into which the maximum and minimum value collection nodes are inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized;

[0141] Step 640: insert quantization and dequantization nodes into the universal intermediate representation according to the obtained maximum and minimum values ​​of the input to be quantized;

[0142] Step 650: Convert the universal intermediate expression with the quantization node and the dequantization node inserted back to a specific format to obtain a quantized model.

[0143] Since the details of the above-mentioned deep learning model quantization method have been described in detail in the introduction of the device part of the above-mentioned embodiment of the present disclosure, they will not be repeated here for the sake of space.

[0144] The commercial value of the disclosed embodiments

[0145] The disclosed embodiment improves the versatility of quantization of the acceleration unit. Compared with adopting different quantization methods for different models, it greatly reduces the quantization deployment cost in practice, making the quantization deployment cost only a fraction of the original cost, and has good market prospects.

[0146] It should be understood that the various embodiments in this specification are described in a progressive manner. References to the same or similar parts of the various embodiments will be sufficient. Each embodiment focuses on the differences from the other embodiments. In particular, the method embodiments are generally similar to the methods described in the device and system embodiments, so their description is relatively simple. For relevant details, references to the descriptions of the other embodiments will suffice.

[0147] It should be understood that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0148] It should be understood that an element described herein in the singular or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown herein as separate may be combined into a single module or element, and modules or elements described or shown herein as single may be split into multiple modules or elements.

[0149] It should also be understood that the terms and expressions used herein are for descriptive purposes only, and the one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or portions thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A processing unit for determining a model suitable for running an acceleration unit in a computing device, reducing requirements for data throughput and storage space, wherein: The processing unit includes: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit, configured to execute the decoded computer instructions to: convert a deep learning model in a specific format into a universal intermediate expression; insert a maximum and minimum value collection node into the universal intermediate expression to collect the maximum and minimum values ​​of the input to be quantized; convert the universal intermediate expression into which the maximum and minimum value collection node has been inserted back into the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; insert quantization and dequantization nodes into the universal intermediate expression based on the obtained maximum and minimum values ​​of the input to be quantized; convert the universal intermediate expression into which the quantization and dequantization nodes have been inserted back into the specific format to obtain a quantized model; and convert the quantized model into an acceleration unit model according to an instruction set supported by the acceleration unit; wherein the acceleration unit is a unit designed to accelerate the computational processing speed of the deep learning model; The processing unit sends the address of the weight of the acceleration unit model in the memory to the acceleration unit in the form of an instruction, so that when the acceleration unit runs the acceleration unit model for inference, the weight is directly retrieved from the memory according to the address and run through the on-chip core of the acceleration unit; the processing unit is also used to convert the input data from the high-precision data type to the low-precision data type during inference, and notify the acceleration unit of the address of the input of the low-precision data type in the memory in the form of an instruction, so that the acceleration unit can use the input data for inference; The step of inserting quantization and dequantization nodes into the universal intermediate expression according to the obtained maximum and minimum values ​​of the input to be quantized includes: Determine the quantization factor based on the obtained maximum and minimum values ​​of the input to be quantized; Generating a quantization node and a dequantization node according to the quantization factor and inserting them into the universal intermediate expression; The quantization node is added before the quantized version of the operation node to quantize the input of the operation node and the weight parameters in the operation node. The dequantization node is added after the quantized version of the operation node to perform dequantization on the output of the operation node.

2. The processing unit according to claim 1, wherein: The quantization factor is configured in response to a user configuration request.

3. The processing unit according to claim 1, wherein: The converting the universal intermediate expression into which the maximum and minimum value collection nodes are inserted back into the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized includes: Converting the general intermediate expression into which the maximum and minimum value collection nodes are inserted back into the deep learning model in the specific format; A calibration dataset is received so as to apply the calibration dataset to a deep learning model converted back into a specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input to be quantized.

4. The processing unit according to claim 3, wherein: The calibration data set is specified in response to a user-specified instruction.

5. The processing unit according to claim 1, wherein The inserting of a maximum and minimum value collection node into the universal intermediate expression includes: Determining an operation node in the universal intermediate expression; A maximum and minimum value collection node is inserted before the operation node selected by the user in the determined operation nodes. The processing unit according to claim 1 , wherein: The instruction execution unit is further configured to execute the decoded computer instructions to implement: after sending the acceleration unit model to the acceleration unit, receiving a quantization accuracy obtained after the acceleration unit runs the acceleration unit model; and determining whether the quantization accuracy meets a predetermined quantization accuracy standard; If not satisfied, redefine the quantitative strategy.

7. The processing unit according to claim 1, wherein: In the deep learning model of the specific format, the operation of the maximum and minimum value collection node is defined and implemented in C++ language.

8. A computing device comprising: The processing unit according to any one of claims 1 to 7; a memory for storing the computer instructions; The acceleration unit is used to run the deployed deep learning model.

9. A system on chip, comprising an acceleration unit for running a deep learning model, wherein the deep learning model is quantized by the processing unit according to any one of claims 1 to 7 and then deployed on the acceleration unit.

10. A data center comprising the computing device according to claim 8.

11. A deep learning model quantization method for determining a model suitable for an acceleration unit in a computing device, reducing requirements for data throughput and storage space, characterized in that: The method comprises: Convert deep learning models in a specific format into a common intermediate representation; Inserting a maximum and minimum value collection node into the general intermediate expression to collect the maximum and minimum values ​​of the input to be quantized; Converting the universal intermediate expression into which the maximum and minimum value collection nodes are inserted back to the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized; Inserting quantization and dequantization nodes into the universal intermediate expression according to the obtained maximum and minimum values ​​of the input to be quantized; Convert the general intermediate expression with the quantization node and the dequantization node inserted back to a specific format to obtain a quantized model; Converting the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit, and sending the model to the acceleration unit; The acceleration unit is designed to accelerate the computing speed of the deep learning model. The processing unit sends the address of the weight of the acceleration unit model in the memory to the acceleration unit in the form of an instruction, so that when the acceleration unit runs the acceleration unit model for inference, the weight is directly retrieved from the memory according to the address and run through the on-chip core of the acceleration unit; the processing unit is also used to convert the input data from the high-precision data type to the low-precision data type during inference, and notify the acceleration unit of the address of the input of the low-precision data type in the memory in the form of an instruction, so that the acceleration unit can use the input data for inference; The step of inserting quantization and dequantization nodes into the universal intermediate expression according to the obtained maximum and minimum values ​​of the input to be quantized includes: Determine the quantization factor based on the obtained maximum and minimum values ​​of the input to be quantized; Generating a quantization node and a dequantization node according to the quantization factor and inserting them into the universal intermediate expression; The quantization node is added before the quantized version of the operation node to quantize the input of the operation node and the weight parameters in the operation node. The dequantization node is added after the quantized version of the operation node to perform dequantization on the output of the operation node.

12. The method according to claim 11, wherein The converting the universal intermediate expression into which the maximum and minimum value collection nodes are inserted back into the specific format for calibration to obtain the maximum and minimum values ​​of the input to be quantized includes: Converting the general intermediate expression into which the maximum and minimum value collection nodes are inserted back into the deep learning model in the specific format; A calibration dataset is received so as to apply the calibration dataset to a deep learning model converted back into a specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input to be quantized.

13. The method according to claim 11, wherein The inserting of a maximum and minimum value collection node into the universal intermediate expression includes: Determining an operation node in the universal intermediate expression; A maximum and minimum value collection node is inserted before the operation node selected by the user in the determined operation nodes.

14. The method according to claim 11, wherein After sending the acceleration unit model to the acceleration unit, the method further includes: Receiving a quantization accuracy obtained after the acceleration unit runs the acceleration unit model; Determining whether the quantization accuracy meets a predetermined quantization accuracy standard; If not satisfied, redefine the quantitative strategy.

Images