Processing unit, related device and method

Abstract

The present disclosure proposes a processing unit, related devices and methods. The processing unit includes: an instruction fetch unit; an instruction decoding unit; an instruction execution unit, which is used to: determine the operator to be quantized based on the static graph of the deep learning model; if the operator to be quantized is a non-composite operator, insert a quantization node in front of the non-composite operator; if the operator to be quantized is a composite operator, determine the atomic operation to be quantized inside the composite operator, and insert a quantization probe in front of the determined atomic operation to be quantized; quantize the deep learning model according to the static graph in which the quantization node is inserted or the quantization probe is inserted in the composite operator. The embodiment of the present disclosure reduces the complexity of the static graph of the deep learning model during quantization, which is beneficial to subsequent optimization and compilation, and avoids the problem that the atomic operations directly decomposed into the composite operator in the static graph are not supported by the framework.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more particularly, to a processing unit, related devices, and methods. Background Art

[0002] In the field of deep learning, inference refers to the application of a pre-trained deep learning model to real-world applications. Regarding inference performance, in addition to hardware-level optimization, model quantization is an important approach to improving inference performance at the algorithmic level. Model quantization involves converting the weights and input data of nodes in a deep learning model from a high-precision quantization level to a low-precision quantization level, for example, converting 32-bit single-precision floating-point numbers (FP32) to 8-bit integer data (INT8). While this reduces accuracy, it helps reduce data throughput and storage requirements.

[0003] In the prior art, quantization is typically achieved by identifying the operators to be quantized in the deep learning model's static graph and adding quantization nodes in front of them. A static graph is a graph generated before code is written based on the deep learning model's structure, representing the relationships between the various operation nodes in the model. Each operation node is represented as an operator in the static graph. Quantization nodes are the programs used to perform quantization. Some operators are relatively simple, consisting of only one or two operations; others are more complex, such as the long short-term memory (LSTM) model and the gated recurrent unit (GRU), which include multiple matrix multiplications and operations such as point additions, point multiplications, and loops between tensors. For such complex operators, simply adding quantization nodes at the operator level to quantize them fails to accurately locate the operations that truly require quantization, resulting in poor quantization accuracy. Therefore, in the prior art, for such complex composite operators, they can first be decomposed into smaller-granularity operators in the static graph. For example, the multiple matrix multiplications and point additions, point multiplications, and loops between tensors contained in the GRU operator are represented as small operators in the static graph. Quantization is then performed at the operator level using the aforementioned insertion of quantization nodes. However, this will introduce many control and loop-related nodes, making the entire static graph complex and hindering subsequent optimization, compilation, and other processes. In addition, some decomposed atomic operations do not exist in the list of operators supported by the framework, and their implementations need to be rewritten, increasing the operator rewriting overhead. Summary of the Invention

[0004] In view of this, the present disclosure aims to reduce the static graph complexity of the deep learning model during quantization, thereby facilitating subsequent optimization and compilation, and avoiding the problem of directly decomposing compound operators into atomic operations in the static graph that are not supported by the framework.

[0005] According to one aspect of the present disclosure, there is provided a processing unit, comprising:

[0006] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0007] an instruction decoding unit for decoding the retrieved computer instructions;

[0008] An instruction execution unit, configured to execute the decoded computer instructions to achieve:

[0009] Determine the operators to be quantized based on the static graph of the deep learning model;

[0010] If the operator to be quantized is a non-composite operator, inserting a quantization node in front of the non-composite operator;

[0011] If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a quantization probe before the determined atomic operation to be quantized;

[0012] The deep learning model is quantized according to a static graph in which a quantization node is inserted or a quantization probe is inserted into a composite operator.

[0013] Optionally, determining the operator to be quantized based on the static graph of the deep learning model includes: converting the static graph of the deep learning model in a specific format into a static graph of a universal intermediate expression; determining the operator to be quantized on the static graph of the universal intermediate expression; quantizing the deep learning model based on the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator includes: converting the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator back to the specific format for quantization to become a quantized model.

[0014] Optionally, after determining the operator to be quantized, the instruction execution unit is further configured to execute the decoded computer instruction to achieve:

[0015] If the operator to be quantized is a non-composite operator, insert a maximum and minimum value collection node in front of the non-composite operator;

[0016] If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a maximum and minimum value collection probe before the determined atomic operation to be quantized;

[0017] Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and into which the maximum and minimum value collection probes are inserted into the composite operator back to the specific format for calibration, and obtaining the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized, respectively;

[0018] The quantization node is determined according to the maximum value and the minimum value of the input of the non-composite operator to be quantized, and the quantization probe is determined according to the maximum value and the minimum value of the input of the atomic operation to be quantized.

[0019] Optionally, determining the operator to be quantized on the static graph of the universal intermediate representation includes: determining the operator to be quantized among the operators of the static graph of the universal intermediate representation according to the accuracy before quantization, the accuracy after quantization and a first rule.

[0020] Optionally, determining the atomic operation to be quantized within the composite operator includes: determining the atomic operation to be quantized within the composite operator according to the accuracy before quantization, the accuracy after quantization, and a second rule.

[0021] Optionally, the quantization node is determined by: determining a quantization factor of the non-composite operator to be quantized according to the maximum value and the minimum value of the input of the non-composite operator to be quantized; and generating a quantization node according to the quantization factor of the non-composite operator to be quantized.

[0022] Optionally, the quantization probe is determined by: determining a quantization factor of the atomic operation to be quantized according to a maximum value and a minimum value of an input of the atomic operation to be quantized; and generating a quantization probe according to the quantization factor of the atomic operation to be quantized.

[0023] Optionally, converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and into which the maximum and minimum value collection probes are inserted into the composite operator back to the specific format for calibration, and obtaining the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized, respectively, includes:

[0024] Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted into the composite operator back into the deep learning model in the specific format;

[0025] A calibration dataset is received so as to apply the calibration dataset to a deep learning model converted back into a specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input of the non-compound operator to be quantized, and the maximum and minimum value collection probe obtains the maximum and minimum values ​​of the input of the atomic operation to be quantized.

[0026] Optionally, after quantizing the deep learning model according to a static graph in which quantization nodes are inserted or quantization probes are inserted into a composite operator, the instruction execution unit is also used to execute the decoded computer instructions to achieve: converting the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit, and sending it to the acceleration unit.

[0027] According to one aspect of the present disclosure, a computing device is provided, comprising: a processing unit as described above; a memory for storing the computer instructions; and an acceleration unit for running a deployed deep learning model.

[0028] According to one aspect of the present disclosure, a system on chip is provided, comprising an acceleration unit for running a deep learning model, wherein the deep learning model is quantized by the processing unit as described above and then deployed on the acceleration unit.

[0029] According to one aspect of the present disclosure, a data center is provided, comprising the computing device described above.

[0030] According to one aspect of the present disclosure, a deep learning model quantization method is provided, comprising:

[0031] Determine the operators to be quantized based on the static graph of the deep learning model;

[0032] If the operator to be quantized is a non-composite operator, inserting a quantization node in front of the non-composite operator;

[0033] If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a quantization probe before the determined atomic operation to be quantized;

[0034] The deep learning model is quantized according to a static graph in which a quantization node is inserted or a quantization probe is inserted into a composite operator.

[0035] Optionally, determining the operator to be quantized based on the static graph of the deep learning model includes: converting the static graph of the deep learning model in a specific format into a static graph of a universal intermediate expression; determining the operator to be quantized on the static graph of the universal intermediate expression; quantizing the deep learning model based on the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator includes: converting the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator back to the specific format for quantization to become a quantized model.

[0036] Optionally, after determining the operator to be quantized, the method further includes:

[0037] If the operator to be quantized is a non-composite operator, insert a maximum and minimum value collection node in front of the non-composite operator;

[0038] If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a maximum and minimum value collection probe before the determined atomic operation to be quantized;

[0039] Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and into which the maximum and minimum value collection probes are inserted into the composite operator back to the specific format for calibration, and obtaining the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized, respectively;

[0040] The quantization node is determined according to the maximum value and the minimum value of the input of the non-composite operator to be quantized, and the quantization probe is determined according to the maximum value and the minimum value of the input of the atomic operation to be quantized.

[0041] Optionally, determining the operator to be quantized on the static graph of the universal intermediate representation includes: determining the operator to be quantized among the operators of the static graph of the universal intermediate representation according to the accuracy before quantization, the accuracy after quantization and a first rule.

[0042] Optionally, determining the atomic operation to be quantized within the composite operator includes: determining the atomic operation to be quantized within the composite operator according to the accuracy before quantization, the accuracy after quantization, and a second rule.

[0043] Optionally, the quantization node is determined by: determining a quantization factor of the non-composite operator to be quantized according to the maximum value and the minimum value of the input of the non-composite operator to be quantized; and generating a quantization node according to the quantization factor of the non-composite operator to be quantized.

[0044] Optionally, the quantization probe is determined by: determining a quantization factor of the atomic operation to be quantized according to a maximum value and a minimum value of an input of the atomic operation to be quantized; and generating a quantization probe according to the quantization factor of the atomic operation to be quantized.

[0045] Optionally, converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and into which the maximum and minimum value collection probes are inserted into the composite operator back to the specific format for calibration, and obtaining the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized, respectively, includes:

[0046] Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted into the composite operator back into the deep learning model in the specific format;

[0047] A calibration dataset is received so as to apply the calibration dataset to a deep learning model converted back into a specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input of the non-compound operator to be quantized, and the maximum and minimum value collection probe obtains the maximum and minimum values ​​of the input of the atomic operation to be quantized.

[0048] In the embodiment of the present disclosure, for non-composite operators to be quantized, quantization is performed by adding a quantization node in front of the non-composite operator as in the prior art. For the composite operator to be quantized, it is not expanded into a small-grained operator in the static graph as in the prior art, but the atomic operation to be quantized is further determined inside the composite operator, and a quantization probe is added in front of it. Then, the deep learning model is quantized based on the static graph in which the quantization node is inserted or the quantization probe is inserted inside the composite operator. In this way, the complex process of splitting the composite operator into multiple atomic operations in the static graph is effectively eliminated, the basic topological structure of the static graph is maintained, and the complexity of the static graph is reduced, which is beneficial to subsequent optimization, compilation and other processes. In addition, since the atomic operations are not decomposed in the static graph, the problem that the decomposed atomic operations do not exist in the list of operators supported by the framework and therefore the operators need to be rewritten is overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0050] Figure 1 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0051] Figure 2 is an internal structure diagram of a server in a data center according to an embodiment of the present disclosure;

[0052] Figure 3 This is an internal structural diagram of a processing unit and an acceleration unit within a server according to an embodiment of the present disclosure;

[0053] Figure 4 is a diagram of the internal structure of an acceleration unit core according to one embodiment of the present disclosure;

[0054] Figure 5 This is a hierarchical architecture diagram of the software environment required to run an exemplary deep learning model;

[0055] Figure 6 The figure shows the overall process of a deep learning model in an embodiment of the present disclosure so that it can be deployed and executed on an acceleration unit;

[0056] Figure 7The model states of the deep learning model at different stages provided by the embodiments of the present disclosure are shown;

[0057] Figure 8 A schematic diagram of inserting a quantization node into a static graph of a deep learning model according to one embodiment of the present disclosure is shown;

[0058] Figure 9 A schematic diagram illustrating inserting a quantization probe into a composite operator according to an embodiment of the present disclosure is shown;

[0059] Figure 10 A flowchart of a deep learning model quantization method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0060] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0061] The following terms are used in this document.

[0062] Deep Learning Model: Deep learning is a new research direction in the field of machine learning (ML). It was introduced to bring ML closer to its original goal: artificial intelligence (AI). Deep learning can learn the inherent patterns and representational hierarchies of sample data. This information gained during learning is highly helpful for interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have human-like analytical learning capabilities and to recognize data such as text, images, and sound. The model used in deep learning is called a deep learning model. Deep learning models have different formats depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet.

[0063] Accelerator units: These are processing units designed to improve data processing speed in specialized applications (e.g., image processing, deep learning model operations, etc.), where traditional processing units are inefficient. Accelerator units, also known as artificial intelligence (AI) processing units, include central processing units (CPUs), graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and specialized intelligent acceleration hardware (e.g., neural network processors (NPUs)).

[0064] Processing unit: A unit within a data center server that performs traditional processing (not the processing performed by the aforementioned acceleration units). In addition to performing these traditional processing tasks, the processing unit also schedules the acceleration units, assigning them tasks. Processing units can take various forms, including CPUs, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0065] Quantization: The act of converting the inputs of operation nodes in a deep learning model, as well as the weight parameters and other parameters of the operation nodes, from high-precision data types (such as 32-bit floating-point numbers) to low-precision data types (such as 8-bit integers), thereby reducing the requirements for data throughput and storage space.

[0066] The following briefly introduces the quantization process with a specific example of quantization.

[0067] During quantization, in order to convert the model parameters and input parameters of the node into low-precision data types, we first need to know the value range of the model parameters and input parameters. For example, the maximum value is 100 and the minimum value is -100. If converted to binary 8-bit integers, let 2 8 -1, i.e. 11111111 corresponds to the maximum value 100, and 0, i.e. 00000000 corresponds to the minimum value -100. At this time, it is equivalent to dividing the total length of the interval between -100 and 100, 200, into (2 8 -1) small intervals, the length of each small interval is 200 / (2 8 -1) = 0.784. Thus, for any value between -100 and 100, its cell interval can be determined. The left or right endpoint of the cell interval is used as the quantized value. For example, for 50, [50 - (-100)] / 0.784 = 191.3, which corresponds to the 191st cell interval, with the left endpoint being 191, or 10111111, and the quantized value being 101111111.

[0068] In the case of linear quantization, quantization can be performed using the following formula. δ is the scaling factor, which is equivalent to the difference between the quantized value and the value before quantization. δ is the maximum value x of the input to be quantized. max , minimum value x min It will be determined by min , x max 】 is quantized to any integer between 0 and n-1 (including 0 and n-1). Let q(x) be the value of the quantized input x, and the formula is:

[0069]

[0070] Among them, clamp is the limit function, and formula 1 means round((xx min ) / δ) is limited to 0 to n-1. If round((xx min ) / δ) is greater than n-1, then the value of formula 1 is n-1; if round((xx min ) / δ) is less than 0, the value of formula 1 is 0; if round((xx min ) / δ) is between 0 and n-1, then the value of Formula 1 is round((xx min ) / δ).

[0071] The scaling factor δ is calculated as follows:

[0072] δ=[max(x max )-min(x min )] / (n-1) Formula 2

[0073] Taking into account the zero point problem (error-free representation of floating point numbers as 0), it is necessary to add an offset to the quantization method, namely z q δ and z q is the maximum value x of the input to be quantized max , minimum value x min decided.

[0074] The scaling factor δ after taking into account the zero point problem is calculated as follows:

[0075]

[0076] Due to the zero point problem, we cannot simply determine which of 0 to n-1 it is quantized based on the value of (x / δ). We must consider the offset z q The offset z q Calculated as follows:

[0077]

[0078] Considering the offset z q Then, the quantized value q(x) is determined according to the pre-quantized value x according to the following formula:

[0079]

[0080] Quantization node: A program and related data inserted into the static graph to execute the above quantization process. The above quantization process is executed after the program is run.

[0081] Dynamic Graphs: Currently, deep learning model frameworks are categorized into static and dynamic graph frameworks. Deep learning models consist of multiple layers, each with multiple operation nodes. Operation nodes are the basic computational units in deep learning models that perform operations based on inputs to produce outputs that are passed to the next layer. The outputs of operation nodes in each layer are transmitted to the operation nodes in the next layer, where they serve as inputs. A deep learning model graph represents each operation node in the model as an operator and the relationships between them as input-output relationships. These graphs are categorized into dynamic and static graphs. A dynamic graph changes with the actual sequence of commands written. This means that each different command generates a different dynamic graph. The required operation nodes and the execution relationships between these nodes are determined based on each command, resulting in a different dynamic graph. This mechanism makes debugging easier and translating ideas into actual code easier. This is because dynamic graphs are generated based on the code after it is written, eliminating the constraints of dynamic graphs during code development. The PyTorch neural network framework utilizes dynamic graphs.

[0082] A static graph is a graph generated before code is written based on the structure of the deep learning model. It represents the various operation nodes in the deep learning model and the relationships between them. Each operation node is represented as an operator in the static graph, and the relationships between them are represented as the input-output relationships of each operator. The deep learning model is defined using a static graph. Thereafter, the graph does not need to be rebuilt each time the deep learning model is run. Therefore, deep learning models run faster using static graphs than dynamic graphs. In theory, mechanisms like static graphs allow the compiler to perform greater optimizations, but this also means a greater gap between the expected program and the actual execution of the compiler. This also means that errors in the code will be more difficult to detect (for example, if there is a problem with the static graph structure, it may only be discovered when the code executes the corresponding operation). Figure 8 The graphic structure on the left, which includes the series-connected convolution operator 620, batch normalization operator 630, activation function operator 640, gated recurrent unit (GRU) operator 650, and fully connected operator 670, is a static diagram of a deep learning model.

[0083] Operator: It is the representation of the operation node in the deep learning model in the static graph. Figure 8 As shown, the static graph of the deep learning model may include a convolution operator 620, a batch normalization operator 630, an activation function operator 640, a GRU operator 650, and a fully connected operator 670. The arrows between the operators represent the input-output relationship of the operators.

[0084] Operators to be quantized: Operators in the static graph whose quantization nodes are to be inserted before them. Quantization aims to reduce data throughput and storage space requirements, but not every operator significantly impacts these requirements. For example, some operators do not store data and therefore do not impact storage space, making their quantization unnecessary. Whether an operator is designated as a node to be quantized also depends on the specific accuracy requirements of the quantization. For example, if quantization requires reducing a high-precision data type to a data type with slightly lower precision, some operators may not need to be quantized; only those with a significant impact on accuracy need to be quantized. However, if quantization requires reducing a very high-precision data type to a very low-precision data type, quantizing only those operators with a significant impact on accuracy may not be sufficient to meet this requirement, and operators with less significant impact on accuracy may also need to be quantized.

[0085] Composite operators: Operators that require multiple instructions to complete, such as the long short-term memory model (LSTM) operator and the gated recurrent unit (GRU) operator. Each of them includes multiple matrix multiplications and point addition, point multiplication, and loop operations between tensors, and therefore requires multiple instructions to complete.

[0086] Non-composite operators: Operators that require a single underlying instruction to complete, such as the dot product of two tensors, such as Figure 8 Convolution operator 620 and fully connected operator 670.

[0087] Atomic operation: The action corresponding to an instruction contained in a composite operator. Since a composite operator requires multiple instructions to complete, the action completed by each instruction can be considered an atomic operation.

[0088] Atomic operations to be quantized: Atomic operations within a composite operator that await the insertion of quantization probes. Quantization aims to reduce data throughput and storage requirements, but not every atomic operation within a composite node significantly impacts these requirements. For example, some atomic operations do not store data and therefore have no impact on storage, making quantization unnecessary. Whether an atomic operation is designated as a quantized atomic operation also depends on the specific precision requirements of quantization. For example, if quantization requires reducing a high-precision data type to a data type with slightly lower precision, some atomic operations may not need to be quantized; only those with a significant impact on precision will be quantized. However, if quantization requires reducing a high-precision data type to a low-precision data type, quantizing only those atomic operations with a significant impact on precision may not be sufficient to meet this requirement, and atomic operations with a less significant impact on precision may also require quantization.

[0089] Quantization probes: Programs and related data inserted within a composite operator to execute the aforementioned quantization process. Once this program is executed, the aforementioned quantization process is performed. Essentially, they are not significantly different from quantization nodes, except that they do not function as nodes themselves and do not disrupt the original static graph structure. In terms of program statements and related data, they are similar.

[0090] Common Intermediate Representation: Deep learning models have different formats, depending on the model framework they rely on, such as TensorFlow, PyTorch, and MXNet. The code representations of these deep learning models also differ. This poses significant challenges to the universality of deep learning model quantification. Common Intermediate Representation (CIR) converts the various deep learning model code representations into a common format. It analyzes the meaning of each code statement in a deep learning model and translates it into a common representation based on its meaning. This ensures that code statements with the same meaning across different deep learning models are expressed identically in the CIR. Currently, there are tools available that convert the representations of different deep learning models into a CIR.

[0091] Quantized model: The model generated by the deep learning model after the above quantization.

[0092] Calibration: As in the quantization example above, quantization first requires determining the value range of the inputs of the operation nodes in the deep learning model. Only by determining the maximum and minimum values ​​of the input to be quantized can the maximum and minimum values ​​of the input to be quantized be mapped to the maximum and minimum values ​​after quantization, thus completing the quantization of the input to be quantized. Calibration is the process of determining the maximum and minimum values ​​of the input to be quantized.

[0093] A Maximum / Minimum Value Collection node is inserted at a specific location in the static graph of a deep learning model. It is used to collect the maximum and minimum values ​​generated at that location during the deep learning model's execution, along with the associated data. This node includes the program code to collect the values ​​generated at that location and determine the maximum and minimum values, as well as the data required for this process. This location is typically before the operator to be quantized. Inserting this node before the operator to be quantized collects the various input values ​​for that operator and determines the maximum and minimum values.

[0094] A Maximum / Minimum Value Collection Probe is inserted at a specific location within a composite operator and is used to collect the maximum and minimum values ​​generated at that location during the deep learning model's execution. This probe includes the program code to collect the values ​​generated at that location and determine the maximum and minimum values, as well as the data required for this process. This location is typically placed before the atomic operation to be quantized. Inserting this node before the atomic operation to be quantized collects the various input values ​​for that operation and obtains the maximum and minimum values. The program code is essentially the same as that of a Maximum / Minimum Value Collection Node, except that it is not inserted into the static graph and does not disrupt its basic structure.

[0095] Pre-quantization precision: The pre-quantization precision of the weight parameters, other parameters, and input data of the corresponding atomic operation within the corresponding operator or compound operator in the static graph. For example, in quantization that converts a 32-bit floating-point number to an 8-bit integer, the pre-quantization precision is 32-bit floating-point.

[0096] Post-quantization precision: The precision of the weight parameters, other parameters, and input data of the corresponding atomic operation within the corresponding operator or compound operator in the static graph after quantization. For example, in quantization that converts a 32-bit floating-point number to an 8-bit integer, the post-quantization precision is an 8-bit integer.

[0097] Quantization factors: When a quantization method is selected, it usually has some factors that are used in the quantization process. These factors are called quantization factors. For example, in the case of linear quantization, the scaling factor and offset are quantization factors.

[0098] Calibration dataset: A dataset used during the calibration process for a deep learning model. Calibration is completed by inputting data from this dataset into the deep learning model and detecting the maximum and minimum values ​​of the input at a specific location within the deep learning model.

[0099] Inference script: A code snippet and related data used to run a deep learning model for actual inference.

[0100] Accelerator unit model: A model converted from a deep learning model that is supported by the acceleration unit's instruction set. A deep learning model can only be run on an acceleration unit after it has been converted to an acceleration unit model.

[0101] Application environment of the present disclosure

[0102] The embodiment of the present disclosure proposes a deep learning model quantization scheme. The entire quantization scheme is relatively universal. The model after quantization on the processing unit can be used in various hardware devices that execute the model, for example, for data centers, for AI processing units, for IOT (Internet of Things) devices that can execute deep learning models, embedded devices, etc. The quantization method is independent of the hardware on which the model is ultimately deployed. However, for the sake of exemplary description, the following description will mainly focus on the data center as the application scenario. Those skilled in the art should understand that the embodiment of the present disclosure can also be applied to other application scenarios.

[0103] Data Center

[0104] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's network infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Deep learning, a key AI technology, has been widely applied to big data analytics in data centers.

[0105] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in Figure 1, this is the hierarchical inter-networking model. This model consists of the following parts:

[0106] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0107] Access switch 130: Access switches 130 connect servers 140 to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, and are therefore also called top-of-rack switches. They physically connect to the servers.

[0108] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0109] Core switch 110: Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for aggregation switch 120. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0110] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0111] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with other aggregation switches 120 being used only in the event of a failure (dashed lines in the figure above). This means that horizontal scalability is impossible at the aggregation switch 120 level because even if multiple aggregation switches 120 are added, only one is still operational.

[0112] server

[0113] Since the server 140 is the real processing device in the data center, Figure 2 The figure shows a block diagram of the internal structure of a server 140. The server 140 includes a memory 210 connected by a bus, a processing unit cluster 270 and an acceleration unit cluster 280. The processing unit cluster 270 includes a plurality of processing units 220. The acceleration unit cluster 280 includes a plurality of acceleration units 230. The acceleration unit 230 is a processing unit designed to increase the data processing speed in a specific application field. The acceleration unit is also called an artificial intelligence (AI) processing unit, and includes a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated intelligent acceleration hardware (for example, a neural network processor NPU). The processing unit is a processing unit that schedules the acceleration unit and allocates a sequence of instructions to be executed to each acceleration unit. It can take various forms such as a processing unit (CPU), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0114] The architectural design of the traditional processing unit makes the control unit and storage unit occupy a large part of the space in the architecture, while the space occupied by the computing unit is insufficient. Therefore, it is very effective in terms of logical control, but not efficient enough in terms of large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively improve the computing speed for calculations of different functions and different fields. The acceleration unit proposed in the present disclosure is a processing unit dedicated to accelerating the computing speed of deep learning models. It is a processing unit that uses a data-driven parallel computing architecture to process a large number of operations (such as convolution, pooling, etc.) of each deep learning model node. Since the data and intermediate results of the large number of operations (such as convolution, pooling, etc.) of each deep learning model node are closely related in the entire computing process and will be used frequently, using the existing processing unit architecture, since the memory capacity within the core of the processing unit is very small, a large number of external core memories must be accessed frequently, resulting in inefficient processing. By using this acceleration unit specifically designed to accelerate the computing processing speed of deep learning models, each core of the unit has on-chip memory with a storage capacity suitable for deep learning model calculations, thereby avoiding frequent access to memory outside the core, it can greatly improve processing efficiency and computing performance.

[0115] The acceleration unit 230 must accept the scheduling of the processing unit 220. Figure 2 As shown, the memory 210 stores various deep learning models, including the nodes of these models and the weight data of the nodes. These deep learning models are used when needed. Figure 2 A processing unit 220 is deployed to an acceleration unit 230. That is, the processing unit 220 can send the address of the parameters in the model (such as the weights of each node) in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually uses the deep learning model for calculation, it will directly address these parameters in the memory 210 according to the addresses of these parameters (such as weights) in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the deep learning model for calculation, the processing unit 220 will also send the input of the model to the acceleration unit 230 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these inputs and the parameters in the model (such as weights). The embodiment of the present disclosure mainly focuses on the quantization process before the processing unit 220 deploys the deep learning model to the acceleration unit 230. Quantization will be described in detail later.

[0116] Internal structure of the processing unit and acceleration unit

[0117] The following combination Figure 3The internal structure diagram of the processing unit 220 and the acceleration unit 230 specifically illustrates how the processing unit 220 schedules the acceleration unit 230 to work.

[0118] like Figure 3 As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0119] The instruction fetch unit 223 is used to move the instruction to be executed from the memory 210 to the instruction register (which may be Figure 3 The register file 229 shown is used to store a register for instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0120] After fetching the instruction, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand fetch information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 226. The operand fetch information may point to an immediate value, a register, or other software / hardware that can provide a source operand.

[0121] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions to efficiently distribute each instruction to different instruction executing units 226 , thereby enabling parallel operation of multiple instructions.

[0122] After the instruction issuing unit 225 sends the instruction to the instruction executing unit 226, the instruction executing unit 226 begins executing the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the instruction is an instruction for deep learning model inference, the instruction executing unit 226 will no longer execute the instruction. Instead, it will send the instruction to the acceleration unit 230 via the bus for execution.

[0123] Figure 3The acceleration unit 30 shown is not limited to the NPU, but can also be a GPGPU. GPGPU, or general-purpose graphics processing unit, is a graphics processor that uses graphics processing tasks to calculate general-purpose computing tasks originally handled by the central processing unit. These general-purpose calculations often have nothing to do with graphics processing. Due to the powerful parallel processing capabilities and programmable pipelines of modern graphics processors, stream processors can process non-graphic data. Especially when faced with single instruction stream multiple data streams (SIMD), and the amount of data processing operations is far greater than the needs of data scheduling and transmission, the general-purpose graphics processor greatly surpasses the traditional central processing unit application in performance, and therefore can play an accelerating role, and belongs to the acceleration unit 30 like the NPU. In addition, the acceleration unit 30 can also include a CPU, a GPU, an FPGA, an ASIC, etc.

[0124] The general structure of the acceleration unit 30 is as follows Figure 3 , which includes multiple cores 236 ( Figure 3 Four cores are shown in the figure, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0125] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0126] Direct Memory Access (DMA) mechanism 235 is a feature provided by some computer bus architectures that enables data to be written directly from an attached device to the computer's mainboard memory. This greatly improves data access efficiency compared to a method where all data transfers between devices must pass through the processing unit. Because of this mechanism, the cores of the acceleration unit 230 can directly access memory 210 and read parameters (such as the weights of each node) in the deep learning model, greatly improving data access efficiency.

[0127] The command processor 237 assigns instructions sent from the processing unit 220 to the acceleration unit 230 to be executed by the core 236. The instruction execution unit 226 sends the acceleration unit 230 a sequence of pending instructions to be executed. After entering the bus channel 231, the pending instruction sequence is cached in the command processor 237, which then selects a core 236 and assigns the instruction sequence to it for execution. The command processor 237 also manages synchronization between the cores 236.

[0128] Acceleration unit core

[0129] Figure 4 FIG. 4 is a diagram of the internal structure of an acceleration unit core according to an embodiment of the present disclosure.

[0130] In one embodiment, Figure 4 As shown, the acceleration unit core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0131] The instruction sequence assigned by the command processor 237 to the acceleration unit core 236 is first cached in the instruction cache 340. The sequencer 350 then fetches instructions from the instruction cache 340 in first-in, first-out order and assigns them to the tensor engine 310 or the pooling engine 320 for execution based on the nature of the instruction. The tensor engine 310 is responsible for processing operations such as convolution and matrix multiplication in deep learning models. The pooling engine 320 is responsible for processing pooling operations in deep learning models. The memory copy engine 330 is a unit specifically responsible for data copying. This data copying involves copying some data from the on-chip memory 360 to the shared memory of each core 236, or to the on-chip memory 360 of other cores 236, due to potential overflow of the on-chip memory 360. The sequencer 350 determines whether to assign the instruction to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 based on the nature of the operation, such as convolution, matrix multiplication, pooling, or data copy.

[0132] The on-chip memory 360 is an in-core memory that stores the weight parameters in the deep learning model, as well as the input parameters and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning model (for example, hyperparameters in the neural network model) in addition to the weight parameters. As described above, in the process of the processing unit 220 pre-configuring the deep learning model in the acceleration unit 230, the processing unit 220 sends the address of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the acceleration unit 230 takes it out from the corresponding position of the memory 210 when the actual deep learning model is operated and places it in the on-chip memory 360. For other parameters, the acceleration unit 230 takes it out from the corresponding position of the memory 210 when the actual deep learning model is operated and places it in the constant buffer 370. Furthermore, when the instruction to actually start inference is assigned by the command processor 237 to the core 236 for execution, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. Furthermore, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0133] Quantitative Operations

[0134] Figure 1-4Shows the hardware environment that the deep learning model depends on. Figure 5 The software architecture that the deep learning model relies on is shown in the figure. As shown in the figure, from top to bottom, the architecture includes application layer 401, framework layer 402, and function layer.

[0135] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, and recommendation 407. These applications are built using this architecture and can also call the architecture to provide runtime interfaces to gain reasoning capabilities within the application.

[0136] The framework layer 402 integrates open-source platforms such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to enable continued optimization and improvement of various algorithms. TensorFlow 408 is a symbolic mathematical system based on data flow programming and is widely used to implement various machine learning algorithms. MXNet 409 is the deep learning library of choice for Amazon. Caffe 410, or Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modularity.

[0137] The functional layer includes a compilation stack 403 and a runtime stack 404. The compilation stack 403 is used for various model conversions (converter 411), quantization 412, optimization 413, and compilation 414. Conversion 411 converts the model's internal data into a common intermediate representation (IR) format. Quantization 412 converts parameters such as weights in deep learning models and inputs to deep learning models from high-precision data types to low-precision data types. Optimization 413 fuses operators within the model and performs operations such as multi-model optimization linking. Compilation 414 optimizes the model based on hardware (e.g., a neural network processor) to generate a binary model that the hardware can recognize. The runtime stack 404 includes a runtime API 415, an execution manager 416, a user-mode driver 417, and a kernel-mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized runtime API 415 provides various runtime callable interfaces. The user-mode driver 417 provides kernel-mode hardware commands and resource scheduling. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0138] The quantization operation referred to herein is the quantization 412 mentioned above, which mainly includes converting parameters such as weights in the deep learning model and the input to the deep learning model from high-precision data types to low-precision data types. This is explained below using a convolutional neural network as an example. A convolutional neural network includes multiple layers (convolutional layers, pooling layers, etc.), each of which has multiple operation nodes. An operation node is the smallest computing unit in a deep learning model that performs operations based on the input to obtain the output passed to the next layer. The features output by the nodes of each convolutional layer are transmitted to the nodes of the next convolutional layer as the input received by the next convolutional layer. Each layer obtains a feature map by performing matrix multiplication and summation calculation using the convolution kernel. The convolution kernel is a two-dimensional matrix, and each value in the two-dimensional matrix can be regarded as a weight parameter. Since the convolutional neural network has multiple layers, there are multiple weight parameters in the convolution kernel of each layer, so the number of weight parameters is huge. Corresponding to the weight parameters, the number of input parameters in the input matrix multiplied by the convolution kernel is also huge. During the model training phase, weight parameters are generally stored and calculated using 32-bit floating-point numbers. The quantization operation is to convert the weight parameters from 32-bit floating-point numbers to a lower-precision data type for storage and calculation, such as converting them to 8-bit integers (signed integers or unsigned integers) or 16-bit floating-point types for storage and calculation. The quantization operation can only convert a portion of the weight parameters, for example, only converting the convolution kernels of a portion of the convolution layer, while the convolution kernels of the remaining convolution layers remain unchanged. Corresponding to the weight parameters, the quantization operation also converts each parameter in the actual input matrix to a lower-precision data type for storage and calculation. The processing unit 220 usually deploys the quantized deep learning model to the acceleration unit 230 in the form of instructions only after completing the quantization operation. That is, the processing unit 220 converts the weights of the deep learning model stored in the memory 210 from a high-precision data type to a low-precision data type, thereby improving data throughput and saving storage space. Then, the processing unit 220 sends the addresses of these quantized weights in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually runs the deep learning model for inference, it can directly retrieve the data from the memory 210 according to these addresses and put it into the on-chip memory 360 of the acceleration unit 230 for execution. At the same time, when the deep learning model is actually running, the processing unit 220 also converts the quantized model actual input from a high-precision data type to a low-precision data type and notifies the acceleration unit 230 of their addresses in the memory 210 in the form of instructions, so that the acceleration unit 230 can use these inputs and weight parameters to run the model, thereby improving data throughput and saving storage space.

[0139] Figure 6 The figure shows the overall process of a deep learning model in an embodiment of the present disclosure so that it can be deployed and executed on an acceleration unit.

[0140] Because deep learning models are written in different frameworks, such as TensorFlow 408, MXNet 409, and Caffe 410, quantization for these different frameworks or formats requires separate quantization methods adapted to each format. To unify subsequent quantization processes, a conversion 411 process is implemented. This involves converting various framework formats, such as TensorFlow 408, MXNet 409, and Caffe 410, into a common intermediate representation (IR) that is independent of any particular framework or format. Quantization is performed on this IR and then converted back to the original framework or format, achieving unified quantization across different frameworks.

[0141] Then, the converted deep learning model is quantized 412, and the weight parameters in the deep learning model and the input parameters to the deep learning model are converted from high-precision data types to low-precision data types, thereby reducing the storage space required by the model and improving the model operation efficiency. The specific process of quantization is as follows: in the general intermediate expression converted by the above conversion process 411, nodes to be quantized and atomic operations to be quantized are determined 4121, wherein composite nodes and non-composite nodes to be quantized are determined in the static graph of the general intermediate expression. For composite nodes to be quantized, since they are composed of multiple atomic operations, the atomic operations to be quantized that need to be quantized must also be accurately determined; before the determined nodes to be quantized and atomic operations to be quantized, maximum and minimum value collection nodes and maximum and minimum value collection probes for collecting the maximum and minimum values ​​of the inputs to be quantized are respectively inserted; the general intermediate expression with the inserted maximum and minimum value collection nodes and probes is converted back to the specific format before the conversion 411 (such as TensorFlow 408) for calibration 4122 to obtain the maximum and minimum values ​​of the inputs to be quantized; based on the obtained maximum and minimum values ​​of the inputs to be quantized, quantization nodes are inserted before the non-composite nodes to be quantized in the general intermediate expression, and quantization probes are inserted before the atomic operations to be quantized of the composite nodes; the general intermediate expression with the inserted quantization nodes and quantization probes is converted back to the specific format before the conversion 411, and quantization is performed 4123 to obtain the quantized model. The detailed process of steps 4121-4123 is the process that the embodiment of the present disclosure mainly focuses on and will be described in more detail below.

[0142] Finally, the quantized deep learning model is compiled 414 into an acceleration unit model that can be recognized by the acceleration unit 230. That is, the processing unit 220 converts the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit 230 and sends it to the acceleration unit 230.

[0143] It should be noted that the above-mentioned conversion 411, quantization 412, and compilation 414 processes are all performed in the processing unit 220. The above process generates an acceleration unit model. After the processing unit 220 deploys the acceleration unit model in the acceleration unit 230, the actual inference process is performed by the acceleration unit 230. That is, the acceleration unit 230 loads the weight parameters in the acceleration unit model into the on-chip memory 360 of the core 236, and also loads the input parameters in the received inference instruction into the on-chip memory 360 of the core 236. The core 236 performs operations based on the weight parameters and input parameters, and finally obtains the running results of the deep learning model.

[0144] Figure 7 The model states of the deep learning model at different stages provided by the embodiment of the present disclosure are shown. Figure 7 As shown, original model 501 is a pre-trained high-precision deep learning model, and quantized model 502 is the deep learning model obtained by quantizing original model 501. The quantization operation can be performed according to steps 4121-4123 above to obtain quantized model 502. Original model 501 is a high-precision model, while quantized model 502 is a low-precision model. Quantized model 502 is then converted into acceleration unit model 503. Acceleration unit model 503 is still a low-precision model. However, acceleration unit model 503 can be recognized by the instruction set of acceleration unit 230 and can be executed on acceleration unit 230, while original model 501 and quantized model 502 can only be recognized by the instruction set of processing unit 220 and can be executed on processing unit 220.

[0145] It should be pointed out that Figure 7 It is only used to illustrate the execution status of the above model, but cannot be used to illustrate the storage status of the above model. In fact, the original model 501, the quantized model 502 and the acceleration unit model 503 are all stored in the memory 210, and will only be loaded into the processing unit 220 or the acceleration unit 230 during execution.

[0146] It should also be pointed out that the model commonly referred to actually contains code and data samples that implement the corresponding algorithm. Code exists in many forms, such as source code (such as Java, Python implementation) and executable code (binary code). The source code can only be recognized and executed by the processor after it is compiled into executable code. Therefore, the above-mentioned conversion of the quantization model 502 into the acceleration unit model 503, so as to be recognized by the instruction set of the acceleration unit 230, refers to modifying part of the source code of the quantization model and compiling it into executable code, and then the executable code can be recognized by the instruction set of the acceleration unit 230, and so on. However, in order to simplify the description, such a strict description will not be performed in each step in this article.

[0147] The quantization method of the embodiment of the present disclosure

[0148] In the existing technology, quantization is generally achieved by determining the operators to be quantized in the static graph of the deep learning model and adding quantization nodes in front. The static graph refers to a graph that is generated in advance before writing the code based on the structure of the deep learning model and represents the relationship between the operation nodes in the deep learning model, such as Figure 8 As shown on the left. Each operation node is represented as an operator in the static graph, such as Figure 8 The convolution operator 620, batch normalization operator 630, activation function operator 640, GRU operator 650, and fully connected operator 670 are shown in FIG. Some operators are relatively simple, including only one or two operations, i.e., non-composite operators, such as the convolution operator 620 and the fully connected operator 670, while some operators are relatively complex, i.e., composite operators, such as the GRU operator 650, which includes multiple matrix multiplications and atomic operations such as dot addition, dot multiplication, and loop between tensors. For composite operators, since they contain multiple atomic operations, it is necessary to accurately determine which atomic operations should be quantized. In the prior art, in order to accurately find the quantization position, the composite operator is decomposed into atomic operations in the static graph, and each atomic operation becomes a new small operator (for example, the multiple matrix multiplications and dot addition, dot multiplication, and loop between tensors contained in the above-mentioned GRU operator are all reflected as small operators in the static graph). In this way, the quantization position can be accurately found on the refined static graph. However, this will introduce many control and loop-related nodes, making the entire static graph complex and hindering subsequent optimization, compilation, and other processes. In addition, some decomposed atomic operations do not exist in the list of operators supported by the framework, and their implementations need to be rewritten, increasing the operator rewriting overhead.

[0149] The embodiment of the present disclosure still adopts the same method of adding quantization nodes in front of non-composite operators to be quantized as in the prior art for quantization. For the composite operators to be quantized, they are not expanded into small-grained operators in the static graph as in the prior art, but the basic structure of the static graph is not destroyed. The atomic operations to be quantized are further determined inside the composite operator, and quantization probes are added in front of them. Then, the deep learning model is quantized based on the static graph in which the quantization nodes are inserted before the non-composite operators to be quantized, or the quantization probes are inserted before the atomic operations to be quantized inside the composite operator to be quantized. In this way, the basic topological structure of the static graph is maintained, and the complexity of the static graph is reduced, which is beneficial to subsequent optimization, compilation and other processes. In addition, since the atomic operations are not decomposed in the static graph, the overhead of rewriting the operators because the decomposed atomic operations do not exist in the operator list supported by the framework is reduced.

[0150] In the embodiment of the present disclosure, the processing unit 220 first determines the operator to be quantized based on the static graph of the deep learning model. The static graph includes operators corresponding to the operation nodes of the deep learning model. Figure 8As shown, the static graph 600 includes a convolution operator 620 corresponding to a convolution operation node, a batch normalization operator 630 corresponding to a batch normalization operation node, an activation function operator 640 corresponding to an activation function operation node, a GRU operator 650 corresponding to a GRU operation node, and a fully connected operator 670 corresponding to a fully connected operation node. The arrows between the operators represent the input-output relationship of the corresponding operation nodes. The operator pointed to by the arrow is the operator of the input data, and the operator led out of the arrow is the operator of the output data. For example, the arrow from the convolution operator 620 to the batch normalization operator 630 indicates that the output of the convolution operator 620 is used as the input of the batch normalization operator 630.

[0151] When determining the operators to be quantized based on the static graph of the deep learning model, the static graph of the deep learning model in a specific format can be first converted into a static graph of a universal intermediate expression, and then the operators to be quantized can be determined on the static graph of the universal intermediate expression. The conversion to a universal intermediate expression is because: as mentioned above, deep learning models are written in different frameworks, such as TensorFlow408, MXNet 409, Caffe 410, etc., with different formats, and their quantization needs to be adapted to these different formats separately. In order to unify the quantization process, various framework formats are converted into a universal intermediate expression, quantized on the basis of the universal intermediate expression, and then converted back to the original framework or format, so as to achieve the effect of unified quantization under different frameworks. This process is Figure 5 and 6 The conversion 411 in can be implemented by using a known conversion method.

[0152] As described above, whether an operator is selected as a node to be quantized depends on the specific precision requirements of the quantization. For example, if quantization requires reducing a high-precision data type to a data type with slightly lower precision, some operators may not need to be quantized; only those with a significant impact on precision need to be quantized. However, if quantization requires reducing a very high-precision data type to a very low-precision data type, quantizing only those operators with a significant impact on precision may not be sufficient to meet this requirement; operators with less significant impact on precision may also require quantization. Therefore, in one embodiment, among the operators in the static graph of the universal intermediate representation, the operators to be quantized are determined based on the pre-quantization precision, the post-quantization precision, and a first rule. The first rule defines a mapping relationship between pre-quantization precision, post-quantization precision, and various operators to be quantized. It clearly specifies which types of operators require quantization under different combinations of pre-quantization precision and post-quantization precision. In actual use, this mapping relationship can be searched based on the desired pre-quantization precision and post-quantization precision to obtain the type of operator to be quantized under that combination of pre-quantization precision and post-quantization precision. Search all operators in the static graph of the general intermediate representation for operators that meet the type of operators to be quantized, i.e., operators to be quantized. Assuming that the precision before quantization is 32-bit floating point and the precision after quantization is 8-bit integer, the types of operators to be quantized corresponding to the combination of 32-bit floating point and 8-bit integer in the mapping relationship include convolution, pooling, LSTM, GRU, and fully connected operators. Figure 8 The operators that meet these types include the convolution operator 620, the GRU operator 650 and the fully connected operator 670, which are used as operators to be quantized.

[0153] Next, determine whether these operators to be quantified are composite operators or non-composite operators. This can be determined by searching the composite operator type library. The composite operator type library contains all composite operator types. If the type of the operator to be quantified belongs to the composite operator type library, then the operator to be quantified is considered a composite operator. Otherwise, it is considered a non-composite operator. Figure 8 As shown, in the node to be quantized, the convolution operator 620 and the fully connected operator 670 are non-composite operators, and the GRU operator 650 is a composite operator.

[0154] If the operator to be quantized is a non-composite operator, a quantization node is inserted before the non-composite operator. The quantization node is a program and related data inserted in the static graph for executing the quantization process. The quantization process is executed after the program is run. Figure 8 As shown, a quantization node 610 is inserted before the convolution operator 620 , and a fully connected operator is inserted before the fully connected operator 670 .

[0155] For a composite operator, since it contains multiple atomic operations, it is necessary to find the atomic operations that actually need to be quantized among these atomic operations. Therefore, it is necessary to determine the atomic operations to be quantized within the composite operator and insert a quantization probe in front of the determined atomic operations to be quantized.

[0156] Whether an atomic operation is used as an atomic operation to be quantized is related to the specific accuracy requirements of the quantization. For example, if quantization needs to reduce a data type with higher precision to a data type with slightly lower precision, some atomic operations may not need to participate in quantization, and only those atomic operations that have a large impact on precision need to be quantized. However, if quantization needs to reduce a data type with very high precision to a data type with very low precision, quantizing only those atomic operations that have a large impact on precision may not be sufficient to meet such requirements, and atomic operations that do not have such a large impact on precision may also need to be quantized. Therefore, in one embodiment, among the atomic operations inside the composite operator, the atomic operations to be quantized are determined based on the precision before quantization, the precision after quantization, and the second rule.

[0157] For compound operators, multiple instructions are required to complete the operation. The action completed by each instruction can be regarded as an atomic operation. Figure 9 , the GRU operator 650 can be represented as a series of atomic operations 1-4, where the output of atomic operation 1 is fed into atomic operation 2 as input, the output of atomic operation 2 is fed into atomic operation 3 as input, and so on.

[0158] The second rule gives the mapping relationship between the accuracy before quantization, the accuracy after quantization and the various types of atomic operations to be quantized. It clearly gives which types of atomic operations need to be quantized under different combinations of various accuracy before quantization and accuracy after quantization. In actual use, the mapping relationship can be searched according to the accuracy before quantization and accuracy after quantization that need to be achieved, and the type of atomic operation to be quantized under the combination of the accuracy before quantization and the accuracy after quantization can be obtained. The operator that meets the type of atomic operation to be quantized, that is, the atomic operation to be quantized, is searched among all the operators in the static graph of the general intermediate expression. Figure 9 As shown, the atomic operations that meet the found type are atomic operations 1 and 3, which are used as operators to be quantized, while atomic operations 2 and 4 are not used as operators to be quantized.

[0159] Then, a quantization probe is inserted before the atomic operation to be quantized. A quantization probe refers to a program and related data inserted inside the composite operator to be quantized for executing the above quantization. After the program is run, the above quantization process is executed. It is essentially not much different from a quantization node, except that it does not act as a node itself and does not destroy the original static graph structure. From the perspective of program statements and related data, it is similar. Figure 9 As shown, a quantization probe 653 is inserted before atomic operations 1 and 3, so that Figure 8 The GRU operator 650 in becomes the quantized GRU operator 651.

[0160] The above embodiment only generally introduces the insertion of quantization nodes in front of non-composite operators and the insertion of quantization probes in front of atomic operations to be quantized. However, in fact, Figure 6 As shown, this quantization is actually completed in two steps: calibration 4122 and quantization 4123 based on the maximum and minimum values ​​obtained by the calibration.

[0161] In calibration 4122, if the operator to be quantized is a non-composite operator, a maximum and minimum value collection node is inserted in front of the non-composite operator. Figure 8 For example, a maximum and minimum value collection node can be inserted in front of the convolution operator 620 and the full connection operator 670, which is similar to Figure 8 If the operator to be quantized is a composite operator, a maximum and minimum value collection probe is inserted before the determined atomic operation to be quantized. Figure 9 For example, we can insert the maximum and minimum value collection probes before atomic operations 1 and 3, which are similar to Figure 9 The positions of the quantified probe 653 insertions.

[0162] A Max / Min Collection node is inserted at a specific location in the static graph of a deep learning model to collect the maximum and minimum values ​​and related data generated at that location during the deep learning model's execution. A Max / Min Collection Probe is inserted at a specific location within a composite operator to collect the maximum and minimum values ​​and related data generated at that location during the deep learning model's execution. The program code is essentially the same as a Max / Min Collection node, except that it is not inserted into the static graph and does not disrupt the underlying structure of the static graph.

[0163] After inserting the maximum and minimum value collection nodes and the maximum and minimum value collection probes, the static graph of the universal intermediate expression in which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted in the composite operator is converted back to the specific format (i.e., the format under the original framework of the deep learning model) for calibration, and the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized are obtained respectively. The calibration is converted back to the original format because the calibration data sets used for calibration are all data sets under the original framework, and the inference script is also a script under the original framework, which does not support the universal intermediate format. Therefore, although the maximum and minimum value collection nodes and the maximum and minimum value collection probes are inserted in the universal intermediate format, they must be converted back to the original format for calibration, that is, to collect the maximum and minimum values.

[0164] After converting the static graph of the universal intermediate representation, which includes the maximum and minimum value collection nodes and the maximum and minimum value collection probes inserted within the composite operators to be quantized, back to the deep learning model in the specific format, a calibration dataset is received so that the calibration dataset can be applied to the converted deep learning model in the specific format using an inference script. The calibration dataset is the dataset applied to the deep learning model during the calibration process. After inputting data from this dataset into the deep learning model, the maximum and minimum values ​​of the input at a specific location in the deep learning model are detected to complete the calibration. The inference script is the code segment and related data used to run the deep learning model for actual inference. The inference script actually contains the program code of the deep learning model. After receiving the calibration dataset, the inference script is executed to obtain the inference results of the deep learning model on the calibration dataset. However, during the calibration process, the model output is not of interest; instead, the maximum and minimum values ​​of the values ​​obtained at the maximum and minimum value collection node locations and the maximum and minimum values ​​of the values ​​obtained at the maximum and minimum value collection probe locations are of interest. These values ​​represent the maximum and minimum values ​​of the inputs of the non-composite operators to be quantized, and the maximum and minimum values ​​of the inputs of the atomic operations to be quantized, respectively. Only when the maximum and minimum values ​​are known can the non-compound operators to be quantized and the atomic operations to be quantized be quantized, that is, the quantization operators inserted before the non-compound operators to be quantized and the quantization probes inserted before the atomic operations to be quantized be generated.

[0165] The quantization node is generated based on the maximum and minimum input values ​​of the non-composite operator to be quantized. Specifically, a quantization factor of the non-composite operator to be quantized can be determined based on the maximum and minimum input values ​​of the non-composite operator to be quantized. Then, a quantization node is generated based on the quantization factor of the non-composite operator to be quantized.

[0166] Quantization factors are factors used in the quantization process. In other words, quantization is determined by these factors, which are determined by the maximum and minimum input values ​​obtained through calibration. The previous description introduced the scaling factor and offset in linear quantization, which are two quantization factors. These two factors primarily control the quantization process. However, in other quantization scenarios, the quantization factors may be other factors.

[0167] The quantization node is a program and related data used to perform the above-mentioned quantization operation. Once the quantization factor is determined, the related program and data are actually determined. Therefore, the quantization node can be generated according to the quantization factor.

[0168] The quantization probe is generated based on the maximum and minimum values ​​of the input of the atomic operation to be quantized. Specifically, a quantization factor of the atomic operation to be quantized can be determined based on the maximum and minimum values ​​of the input of the atomic operation to be quantized. Then, a quantization probe is generated based on the quantization factor of the atomic operation to be quantized.

[0169] The process of determining the quantization factor of the atomic operation to be quantized based on the maximum and minimum values ​​of the input of the atomic operation to be quantized, and generating a quantization probe based on the quantization factor of the atomic operation to be quantized is similar to the process of determining the quantization factor of the non-composite operator to be quantized based on the maximum and minimum values ​​of the input of the non-composite operator to be quantized, and generating a quantization node based on the quantization factor of the non-composite operator to be quantized, so it is not repeated here.

[0170] In the above process, a quantization node has been inserted into the static graph of the general intermediate expression, and a quantization probe has been inserted into the composite operator to be quantized. However, quantization needs to adapt to the original framework of the deep learning model. Therefore, the static graph of the general intermediate expression with the quantization node inserted or the quantization probe inserted into the composite operator to be quantized is converted back to the format of the original framework of the deep learning model for quantization to become a quantized model. Then, the quantized model is converted into an acceleration unit model 503 (i.e., the instruction set supported by the acceleration unit) according to the instruction set supported by the acceleration unit. Figure 5-6 414 in the compilation) and sent to the acceleration unit 230.

[0171] As described above, the embodiment of the present disclosure does not expand the composite operator to be quantized into small-grained operators in the static graph as in the prior art, but further determines the atomic operation to be quantized within the composite operator and adds a quantization probe in front of it. Then, the deep learning model is quantized based on the static graph in which the quantization node is inserted or the quantization probe is inserted within the composite operator to be quantized. In this way, the basic topological structure of the static graph is maintained, and the complexity of the static graph is reduced, which is beneficial to subsequent optimization, compilation, and other processes. At the same time, the risk of the decomposed atomic operation not existing in the list of operators supported by the framework and thus requiring the operator to be rewritten is reduced.

[0172] like Figure 10 As shown, the embodiment of the present disclosure also provides a deep learning model quantization method, which is executed by the processing unit 220. The method includes:

[0173] Step 710: Determine an operator to be quantized based on the static graph of the deep learning model;

[0174] Step 720: If the operator to be quantized is a non-composite operator, insert a quantization node before the non-composite operator;

[0175] Step 730: If the operator to be quantized is a composite operator, determine the atomic operation to be quantized within the composite operator, and insert a quantization probe before the determined atomic operation to be quantized;

[0176] Step 740: quantize the deep learning model according to the static graph in which the quantization node is inserted or the quantization probe is inserted into the composite operator.

[0177] Since the specific implementation process of this method has been involved in the description of the previous device embodiment, it will not be repeated here to save space.

[0178] The commercial value of the disclosed embodiments

[0179] Since the embodiment of the present disclosure omits the complex process of splitting a composite operator into multiple atomic operations in a static graph, the complexity of the static graph is greatly reduced. Experiments have shown that compared with the prior art solution of splitting a composite operator into multiple atomic operations in a static graph, the optimization and compilation speeds are several times faster, and the possibility that the decomposed atomic operations do not exist in the list of operators supported by the framework is eliminated. The deployment efficiency of the model is improved several times, and it has good market prospects.

[0180] It should be understood that the various embodiments in this specification are described in a progressive manner. References to the same or similar parts of the various embodiments will be sufficient. Each embodiment focuses on the differences from the other embodiments. In particular, the method embodiments are generally similar to the methods described in the device and system embodiments, so their description is relatively simple. For relevant details, references to the descriptions of the other embodiments will suffice.

[0181] It should be understood that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0182] It should be understood that an element described herein in the singular or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown herein as separate may be combined into a single module or element, and modules or elements described or shown herein as single may be split into multiple modules or elements.

[0183] It should also be understood that the terms and expressions used herein are for descriptive purposes only, and the one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or portions thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A processing unit for determining a model suitable for running an acceleration unit in a computing device, reducing requirements for data throughput and storage space, wherein: The processing unit includes: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit, configured to execute the decoded computer instructions to achieve: Determine the operators to be quantized based on the static graph of the deep learning model; If the operator to be quantized is a non-composite operator, inserting a quantization node in front of the non-composite operator; If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a quantization probe before the determined atomic operation to be quantized; quantizing the deep learning model according to a static graph into which a quantization node is inserted, or into which a quantization probe is inserted, and converting the quantized model into an acceleration unit model according to an instruction set supported by the acceleration unit; The acceleration unit is designed to accelerate the computing speed of the deep learning model. Among them, the processing unit sends the address of the weight of the acceleration unit model in the memory to the acceleration unit in the form of an instruction, so that when the acceleration unit runs the acceleration unit model for reasoning, the weight is directly taken out from the memory according to the address and run through the on-chip core of the acceleration unit; the processing unit is also used to convert the input data from a high-precision data type to a low-precision data type during reasoning, and notifies the acceleration unit of the address of the input of the low-precision data type in the memory in the form of an instruction, so that the acceleration unit can use the input data for reasoning.

2. The processing unit according to claim 1, wherein: The static graph based on the deep learning model determines the operator to be quantized, including: Convert the static graph of a deep learning model in a specific format into a static graph of a universal intermediate representation; Determining an operator to be quantized on a static graph of the universal intermediate representation; The quantization of the deep learning model is performed based on the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator, including: converting the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator back to the specific format for quantization to become a quantized model.

3. The processing unit according to claim 2, wherein: After determining the operator to be quantized, the instruction execution unit is further configured to execute the decoded computer instructions to achieve: If the operator to be quantized is a non-composite operator, insert a maximum and minimum value collection node in front of the non-composite operator; If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a maximum and minimum value collection probe before the determined atomic operation to be quantized; Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and into which the maximum and minimum value collection probes are inserted into the composite operator back to the specific format for calibration, thereby obtaining the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized, respectively; The quantization node is determined according to the maximum value and the minimum value of the input of the non-composite operator to be quantized, and the quantization probe is determined according to the maximum value and the minimum value of the input of the atomic operation to be quantized.

4. The processing unit according to claim 2, wherein: The determining of the to-be-quantized operator on the static graph of the universal intermediate representation includes: Among the operators of the static graph of the universal intermediate representation, operators to be quantized are determined according to the accuracy before quantization, the accuracy after quantization, and a first rule.

5. The processing unit according to claim 1, wherein Determining the atomic operation to be quantized within the composite operator includes: Inside the composite operator, the atomic operation to be quantized is determined according to the precision before quantization, the precision after quantization, and the second rule. The processing unit according to claim 3 , wherein: The quantization node is determined in the following manner: Determining a quantization factor of the non-composite operator to be quantized according to the maximum value and the minimum value of the input of the non-composite operator to be quantized; A quantization node is generated according to the non-composite operator quantization factor to be quantized.

7. The processing unit according to claim 3, wherein: The quantitative probe is determined as follows: Determining a quantization factor of the atomic operation to be quantized according to the maximum value and the minimum value of the input of the atomic operation to be quantized; A quantization probe is generated according to the quantization factor of the atomic operation to be quantized.

8. The processing unit according to claim 3, wherein: The static graph of the universal intermediate expression in which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted into the composite operator is converted back to the specific format for calibration, and the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized are obtained respectively, including: Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted into the composite operator back into the deep learning model in the specific format; A calibration dataset is received so as to apply the calibration dataset to a deep learning model converted back into a specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input of the non-compound operator to be quantized, and the maximum and minimum value collection probe obtains the maximum and minimum values ​​of the input of the atomic operation to be quantized.

9. A computing device comprising: The processing unit according to any one of claims 1 to 8; a memory for storing the computer instructions; Accelerator units, used to run deployed deep learning models.

10. A system on chip, comprising an acceleration unit for running a deep learning model, wherein the deep learning model is quantized by the processing unit according to any one of claims 1 to 8 and then deployed on the acceleration unit.

11. A data center comprising a server, wherein the server comprises the computing device according to claim 9.

12. A deep learning model quantization method for determining a model suitable for an acceleration unit in a computing device, thereby reducing requirements for data throughput and storage space, the method comprising: Determine the operators to be quantized based on the static graph of the deep learning model; If the operator to be quantized is a non-composite operator, inserting a quantization node in front of the non-composite operator; If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a quantization probe before the determined atomic operation to be quantized; quantizing the deep learning model according to a static graph into which a quantization node is inserted, or a quantization probe is inserted into a composite operator; Convert the quantized model into an acceleration unit model according to the instruction set supported by the acceleration unit; The acceleration unit is designed to accelerate the computing speed of the deep learning model. Among them, the processing unit sends the address of the weight of the acceleration unit model in the memory to the acceleration unit in the form of an instruction, so that when the acceleration unit runs the acceleration unit model for reasoning, the weight is directly taken out from the memory according to the address and run through the on-chip core of the acceleration unit; the processing unit is also used to convert the input data from a high-precision data type to a low-precision data type during reasoning, and notifies the acceleration unit of the address of the input of the low-precision data type in the memory in the form of an instruction, so that the acceleration unit can use the input data for reasoning.

13. The method according to claim 12, wherein: The static graph based on the deep learning model determines the operator to be quantized, including: Convert the static graph of a deep learning model in a specific format into a static graph of a universal intermediate representation; Determining an operator to be quantized on a static graph of the universal intermediate representation; The quantization of the deep learning model is performed based on the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator, including: converting the static graph in which the quantization node is inserted or the quantization probe is inserted into the compound operator back to the specific format for quantization to become a quantized model.

14. The method according to claim 13, wherein: After determining the operator to be quantized, the method further includes: If the operator to be quantized is a non-composite operator, insert a maximum and minimum value collection node in front of the non-composite operator; If the operator to be quantized is a composite operator, determining an atomic operation to be quantized within the composite operator, and inserting a maximum and minimum value collection probe before the determined atomic operation to be quantized; Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and into which the maximum and minimum value collection probes are inserted into the composite operator back to the specific format for calibration, thereby obtaining the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized, respectively; The quantization node is determined according to the maximum value and the minimum value of the input of the non-composite operator to be quantized, and the quantization probe is determined according to the maximum value and the minimum value of the input of the atomic operation to be quantized.

15. The method according to claim 13, wherein The determining of the to-be-quantized operator on the static graph of the universal intermediate representation includes: Among the operators of the static graph of the universal intermediate representation, operators to be quantized are determined according to the accuracy before quantization, the accuracy after quantization, and a first rule.

16. The method according to claim 12, wherein: Determining the atomic operation to be quantized within the composite operator includes: Inside the composite operator, the atomic operation to be quantized is determined according to the precision before quantization, the precision after quantization, and the second rule.

17. The method according to claim 14, wherein: The quantization node is determined in the following manner: Determining a quantization factor of the non-composite operator to be quantized according to the maximum value and the minimum value of the input of the non-composite operator to be quantized; A quantization node is generated according to the non-composite operator quantization factor to be quantized.

18. The method according to claim 14, wherein The quantitative probe is determined as follows: Determining a quantization factor of the atomic operation to be quantized according to the maximum value and the minimum value of the input of the atomic operation to be quantized; A quantization probe is generated according to the quantization factor of the atomic operation to be quantized.

19. The method according to claim 14, wherein The static graph of the universal intermediate expression in which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted into the composite operator is converted back to the specific format for calibration, and the maximum and minimum values ​​of the inputs of the non-composite operator to be quantized and the atomic operation to be quantized are obtained respectively, including: Converting the static graph of the universal intermediate expression into which the maximum and minimum value collection nodes are inserted and the maximum and minimum value collection probes are inserted into the composite operator back into the deep learning model in the specific format; A calibration dataset is received so as to apply the calibration dataset to a deep learning model converted back into a specific format using an inference script, wherein the maximum and minimum value collection node obtains the maximum and minimum values ​​of the input of the non-compound operator to be quantized, and the maximum and minimum value collection probe obtains the maximum and minimum values ​​of the input of the atomic operation to be quantized.

Images