Quantization methods and related devices for deep learning models

Abstract

The present disclosure provides a method and related apparatus for quantizing a deep learning model. The method comprises: receiving a deep learning model to be quantized; dividing the deep learning model to be quantized into sub-models; for each sub-model, selecting a quantization algorithm and a quantization strategy corresponding to the sub-model from a combination of pre-set candidate quantization algorithms and candidate quantization strategies, wherein the quantization strategy is a criterion to be followed in addition to the quantization algorithm during the quantization process; and outputting a quantized deep learning model obtained by quantizing the sub-model according to the corresponding quantization algorithm and quantization strategy. The embodiments of the present disclosure overcome the problems of low precision or high complexity in the prior art model quantization, thereby ensuring quantization precision while reducing complexity.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and more specifically, to a quantization method and related device for a deep learning model. Background Art

[0002] Deep learning models have been widely used in various fields. After generating a deep learning model, it must be deployed for execution on a local chip or in the cloud. The performance of the deployment process directly impacts the performance and user experience when using the deep learning model for inference. Quantization is a critical step in the deployment process.

[0003] Quantization involves converting the weights and input data of each node in a deep learning model from a high-precision quantization level to a lower-precision quantization level, for example, from 32-bit single-precision floating-point numbers (FP32) to 8-bit integer data (INT8). While this reduces precision, it helps reduce data throughput and storage requirements. The quality of the quantization algorithm and strategy directly determines the accuracy and speed of the model ultimately deployed on the chip.

[0004] Existing deployment frameworks typically use a single quantization algorithm and strategy for the entire model. For example, if an asymmetric quantization algorithm is selected for the Mobilenet V1 model and the KL divergence calibration method is chosen, the same algorithm and strategy will be used for quantization of the entire model. This approach can lead to significant time losses during quantization calibration or cause the quantization accuracy of some models to degrade. Summary of the Invention

[0005] In view of this, the present disclosure aims to overcome the problems of low precision or high complexity in model quantization in the prior art, and to achieve both guaranteed quantization accuracy and reduced complexity.

[0006] To achieve this objective, according to one aspect of the present disclosure, the present disclosure provides a quantization method for a deep learning model, comprising:

[0007] Receive the deep learning model to be quantized;

[0008] Divide the deep learning model to be quantized into sub-models;

[0009] For a sub-model, a quantization algorithm and a quantization strategy corresponding to the sub-model are selected from a combination of preset candidate quantization algorithms and candidate quantization strategies, wherein the quantization strategy is a criterion to be followed in addition to the quantization algorithm during the quantization process;

[0010] The output sub-model is quantized according to the corresponding quantization algorithm and quantization strategy to obtain the quantized deep learning model.

[0011] Optionally, before selecting the quantization algorithm and quantization strategy corresponding to the sub-model, the method also includes: receiving a quantization direction input by a user; the selecting of the quantization algorithm and quantization strategy corresponding to the sub-model includes: selecting a quantization algorithm and quantization strategy corresponding to the sub-model and the quantization direction.

[0012] Optionally, the receiving of the quantization direction input by the user includes: receiving the quantization direction input by the user corresponding to the divided sub-model; the selecting of the quantization algorithm and quantization strategy corresponding to the sub-model and the quantization direction includes: selecting the quantization algorithm and quantization strategy according to the sub-model and the quantization direction corresponding to the divided sub-model.

[0013] Optionally, dividing the deep learning model to be quantized into sub-models includes: inputting the deep learning model to be quantized into a sub-model division model, and the sub-model division model outputs the divided sub-models.

[0014] Optionally, dividing the deep learning model to be quantified into sub-models includes: dividing the nodes into sub-models according to predetermined rules based on at least one of the types and levels of the nodes in the deep learning model to be quantified.

[0015] Optionally, dividing the deep learning model to be quantized into sub-models includes:

[0016] Generate and display a static graph corresponding to the deep learning model to be quantized;

[0017] receiving a user's instruction to divide the static graph into sub-models;

[0018] Divide the deep learning model into sub-models according to the sub-model division indication.

[0019] Optionally, the quantization strategy includes at least one of a calibration algorithm and a number of quantized bits.

[0020] Optionally, the combination of the candidate quantization algorithm and the candidate quantization strategy is preset in the following manner:

[0021] Acquire a quantization algorithm compatible with the acceleration unit as a candidate quantization algorithm;

[0022] Acquire a quantization strategy compatible with the acceleration unit as a candidate quantization strategy;

[0023] The candidate quantization algorithms and the candidate quantization strategies are combined with each other.

[0024] Optionally, the selecting the quantization algorithm and quantization strategy corresponding to the sub-model and the quantization direction includes: inputting the sub-model and the quantization direction into a quantization algorithm and strategy selection model, and the quantization algorithm and quantization strategy selection model outputs the selected quantization algorithm and quantization strategy.

[0025] Optionally, selecting a quantization algorithm and a quantization strategy corresponding to the sub-model includes:

[0026] Displaying a combination of the preset candidate quantization algorithms and candidate quantization strategies;

[0027] receiving a user's selection indication from the combination of the preset candidate quantization algorithms and candidate quantization strategies;

[0028] According to the selection instruction, a quantization algorithm and a quantization strategy corresponding to the sub-model are selected.

[0029] Optionally, the selecting of the quantization algorithm and quantization strategy corresponding to the sub-model includes: selecting corresponding quantization algorithms and quantization strategies for the input tensors, weights and offsets of the nodes in the sub-model, respectively.

[0030] Optionally, quantizing the sub-model according to a corresponding quantization algorithm and quantization strategy includes:

[0031] calibrating the nodes of the sub-model according to the calibration algorithm;

[0032] The calibrated nodes are quantized according to the quantization algorithm and the quantized bit number.

[0033] According to one aspect of the present disclosure, a cloud service system is provided, comprising a plurality of cloud servers for distributedly storing computer executable codes, wherein when the computer executable codes are executed by processors on corresponding cloud servers, the method described above is implemented.

[0034] According to one aspect of the present disclosure, a data center is provided, comprising the cloud service system as described above.

[0035] According to one aspect of the present disclosure, there is provided a computer device, comprising:

[0036] a memory for storing computer executable code;

[0037] The processor is configured to execute the computer executable code stored in the memory to implement the method described above.

[0038] In the disclosed embodiments, a deep learning model is divided into sub-models. For each sub-model, an appropriate quantization algorithm and strategy are determined to avoid the problem of applying the same quantization algorithm and strategy to the entire model, which either results in a huge loss of time or an explosion in accuracy. This avoids the quantization algorithm and strategy being sensitive to the model, and also avoids forcibly applying the quantization algorithm and strategy for the entire network model to each sub-model, which increases the computational complexity of quantization. This ensures both quantization accuracy and reduced complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other objects, features and advantages of the present disclosure will become more apparent through description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0040] Figure 1 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0041] Figure 2A -D is an interface change diagram of the application of the quantization method of the deep learning model in the embodiment of the present disclosure in a cloud scenario;

[0042] Figure 3 A schematic diagram illustrating a quantization process according to an embodiment of the present disclosure is shown;

[0043] Figure 4 A flowchart of a deep learning model quantization method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0044] The present disclosure is described below based on examples, but the present disclosure is not limited to these examples. Certain specific details are described in detail in the detailed description of the present disclosure below. Those skilled in the art will appreciate that the present disclosure is fully understood without these details. To avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures have not been described in detail. The accompanying drawings are not necessarily drawn to scale.

[0045] The following terms are used in this document.

[0046] Deep Learning Model: Deep learning is a new research direction in machine learning, introduced to bring machine learning closer to its original goal: artificial intelligence. Deep learning learns the inherent patterns and representational hierarchies of sample data. The information gained during this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound. The model used in deep learning is called a deep learning model.

[0047] Quantization: The act of converting model parameters (such as weights, biases) and input tensors of nodes in a deep learning model from high-precision data types to low-precision data types, thereby reducing the requirements for data throughput and storage space.

[0048] The following briefly introduces the quantization process with a specific example of quantization.

[0049] During quantization, in order to convert the model parameters (weights, offsets) and input tensors of the nodes into low-precision data types, we first need to know the value range of the model parameters and input tensors. For example, the maximum value is 100 and the minimum value is -100. If converted to binary 8-bit integers, let 2 8 -1, i.e. 11111111 corresponds to the maximum value 100, and 0, i.e. 00000000 corresponds to the minimum value -100. At this time, it is equivalent to dividing the total length of the interval between -100 and 100, 200, into (2 8 -1) small intervals, the length of each small interval is 200 / (2 8 -1) = 0.784. Thus, for any value between -100 and 100, its cell interval can be determined. The left or right endpoint of the cell interval is used as the quantized value. For example, for 50, [50 - (-100)] / 0.784 = 191.3, which corresponds to the 191st cell interval, with the left endpoint being 191, or 10111111, and the quantized value being 101111111.

[0050] In the above process, the pre-quantization values in the value range of the pre-quantization values are uniformly distributed in the value range of the post-quantization values after quantization. This quantization is called linear quantization. The common linear quantization process can be expressed by the following mathematical expression:

[0051] r=Round(S(qZ)) Formula 1

[0052] Here, q represents the value before quantization (e.g., a 32-bit floating-point number); Z represents the zero offset of the value before quantization; S represents the quantization scaling factor Scale; Round(·) represents a mathematical function that rounds to an integer. In addition to rounding, rounding up or down is also possible; and r represents the integer value after quantization. In other words, the zero offset Z is subtracted from the value before quantization, and the difference is multiplied by the quantization scaling factor S. After rounding, the quantized value r is obtained. The quantization scaling factor S can be determined by dividing the difference between the maximum and minimum values of the quantized value by the difference between the maximum and minimum values of the value before quantization, such as 0.784 in the above example.

[0053] Linear quantization can be divided into symmetric quantization and asymmetric quantization.

[0054] Symmetrical quantization: The quantization with the zero offset Z as 0, that is, using the above formula 1, the quantized value is located in an interval that is basically symmetrical around the zero point, such as [-2 7 , 2 7 -1]. In the above example of quantizing the pre-quantization value with a maximum value of 100 and a minimum value of -100, the minimum value -100 corresponds to the endpoint -2 7 , the maximum value 100 corresponds to endpoint 2 7 -1.

[0055] A key aspect of symmetric quantization is zero point processing. Mapping Formula 1 ensures that zero points in the original pre-quantized value still correspond to zero points within the quantized value range after passing through Formula 1. In symmetric quantization, r is represented by a signed integer. In this case, when Z = 0 and q = 0, r = 0.

[0056] Asymmetric quantization: quantization in which the zero offset Z is not 0, that is, using the above formula 1, the quantized value is located in an interval whose endpoints are not basically symmetrical around the zero point, such as [0, 2 8 In the above example of quantizing the pre-quantization value with a maximum value of 100 and a minimum value of -100, the minimum value -100 corresponds to the endpoint 0, and the maximum value 100 corresponds to the endpoint 2. 8 -1.

[0057] Node: The smallest unit of independent computation in a deep learning model. It receives an input tensor, performs its own computation, and generates an output tensor, which it then passes to the nodes in the next layer. Deep learning models have multiple layers, each with multiple nodes. The output of each node serves as the input for the nodes in the next layer. For example, a convolution node receives an input tensor, multiplies it by the weights in its convolution kernel, adds the node's offset, and generates an output tensor, which it then passes to the nodes in the next layer.

[0058] Submodel: A portion of a deep learning model where the nodes within it share the same quantization algorithm and strategy. Generally speaking, nodes assigned to the same submodel share certain common properties, such as being the same type of node, such as a convolutional node, or being nodes at the same layer. For example, if you divide submodels 505 by node layer, nodes at the same layer are grouped into the same submodel 505.

[0059] Quantization algorithm: The algorithm used for quantization. Symmetric quantization 511 and asymmetric quantization 512, as described above, are two common quantization algorithms.

[0060] Quantization strategy: Principles that must be followed during the quantization process, in addition to the quantization algorithm. Quantization also requires adherence to other principles. For example, quantization includes a prior calibration process. The calibration algorithm used during calibration is a quantization strategy. Furthermore, in addition to the quantization algorithm, quantization also requires adherence to the number of bits after quantization (for example, all model input tensors and parameters must be quantized to 8-bit integers). The number of bits after quantization is another quantization strategy.

[0061] Candidate quantization algorithms: Candidate quantization algorithms from which the quantization algorithm to be used is selected.

[0062] Candidate quantitative strategies: Candidate quantitative strategies from which the quantitative strategies to be adopted are selected.

[0063] Quantization direction: The goal pursued by quantization. For example, if the goal is a fast quantization process with minimal time consumption, the quantization direction is to improve computational efficiency; if the goal is a simple model after quantization, the quantization direction is to reduce model complexity; if the goal is a small storage space occupied by the quantized model, the quantization direction is to reduce model storage space. Different quantization directions determine the choice of quantization algorithm and quantization strategy.

[0064] Calibration: As mentioned above, quantization first requires determining the value range of the input parameters of the model nodes. The maximum value of the range is then aligned with the quantized maximum value, and the minimum value of the range is aligned with the quantized minimum value, thereby performing quantization. Determining the value range of the input parameters of each node in a deep learning model is called calibration. The input tensors of a node in the deep learning model at several time points can be extracted and the value range of the input tensor of that node can be summarized from the input tensors at these time points. Calibration is the foundation of quantization.

[0065] Sub-model division model: The embodiment of the present disclosure is a neural network model specifically used to divide the deep learning model to be divided into sub-models, and its training process is described in detail later.

[0066] Node type: Since a node is the smallest unit of independent operation in a deep learning model, the type of operation it performs is the type of node. For example, the type of node that performs a convolution operation is a convolution node.

[0067] Node hierarchy: As mentioned above, there are multiple layers in the deep learning model, each layer has multiple nodes, and the output of each node is the input of the node in the next layer. The layers containing multiple nodes in the above deep learning model are called node hierarchies.

[0068] Static graph: This is a graph that shows the nodes in a deep learning model and the relationships between them. It shows the type and level of each node, and uses arrows to represent the signal input and output relationships between nodes.

[0069] Submodel partitioning instructions: Users operate on the static graph to indicate that certain nodes will be partitioned into a submodel. For example, if a user draws a circle on the static graph, the circled nodes will be partitioned into a submodel.

[0070] Quantization algorithm and strategy selection model: The embodiment of the present disclosure is dedicated to outputting a neural network model of the quantization algorithm and quantization strategy selected for the sub-model, and its training process is described in detail later.

[0071] Cloud application scenarios

[0072] Data centers are globally coordinated networks of specialized equipment used to transmit, accelerate, display, compute, and store data on the internet's network infrastructure. In the future, data centers will become a competitive asset for businesses. With the widespread use of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Deep learning, a key AI technology, has been widely applied to big data analytics in data centers.

[0073] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in Figure 1, this is the hierarchical inter-networking model. This model consists of the following parts:

[0074] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0075] Access switch 130: Access switches 130 connect servers 140 to the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are typically located at the top of a rack, and are therefore also called top-of-rack switches. They physically connect to the servers.

[0076] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0077] Core switch 110: Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for aggregation switch 120. The entire data center network is divided into an L3 routing network and an L2 routing network. Core switch 110 generally provides a resilient L3 routing network for the entire data center network.

[0078] Typically, aggregation switch 120 is the demarcation point between the L2 and L3 routing networks. The network below aggregation switch 120 is the L2 network, and the network above it is the L3 network. Each aggregation switch group manages a point of delivery (POD), each of which is an independent VLAN network. Servers can migrate within a POD without changing their IP addresses or default gateways, as each POD corresponds to a single L2 broadcast domain.

[0079] The Spanning Tree Protocol (STP) is typically used between aggregation switches 120 and access switches 130. STP ensures that only one aggregation switch 120 is available for a given VLAN network, with other aggregation switches 120 being used only in the event of a failure (dashed lines in the figure above). This means that horizontal scalability is impossible at the aggregation switch 120 level because even if multiple aggregation switches 120 are added, only one is still operational.

[0080] Example of an application interface for a cloud scenario

[0081] Figure 2A -D is the interface change diagram of the quantization method of the deep learning model of the embodiment of the present disclosure applied in the cloud scenario. In the cloud scenario, the cloud is equivalent to having Figure 1 The data center shown. Among all the servers 140, there is an entry server (not shown). The entry server communicates with the user's client device through the Internet. The interface is displayed on the entry server, allowing the user to send the deep learning model file to be quantized from the client device to the entry server. After receiving it, the entry server may perform the quantization process of the deep learning model of the embodiment of the present disclosure by itself, and send the quantized deep learning model file back to the user's client device, but it is more likely to be distributed to other servers 140 in the cloud to perform part of the above quantization process respectively, and send the quantized deep learning model file back to the user's client device. The client device here can be a desktop computer, tablet computer, PDA, dedicated equipment, vehicle-mounted equipment, mobile terminal, etc. In this cloud scenario, the quantization method of the deep learning model is executed by each server 140 in the cloud, but Figure 2A The -D interface is the interface displayed on the user's client.

[0082] like Figure 2AAs shown, when a user opens the webpage requested from the portal server on the client, the client displays a webpage instructing the user to upload the deep learning model file to be quantized. The deep learning model file is a file containing all the instructions and parameters of the deep learning model. The user can select "Browse" on the interface, choose the path of the locally stored deep learning model file to be quantized, and then select "Upload" to upload the deep learning model file to be quantized.

[0083] like Figure 2B As described above, the entry server receives the deep learning model file to be quantified, and cooperates with other servers 140 in the cloud to convert the deep learning model into a static graph. In the static graph, you can clearly see the nodes of the deep learning model and the signal input and output relationship between the nodes. The static graph is sent to the client for display. The user can divide the sub-model by drawing circles on the interface, and the nodes in each circle constitute a sub-model. After circling, confirm on the interface. If the user does not want to divide the sub-model by himself, he can choose to automatically divide the sub-model on the interface. In this way, the server automatically divides the sub-model according to the method detailed later.

[0084] When Figure 2B As shown, after dividing the sub-model, you can Figure 2C In the client, a user interface is displayed, allowing the user to select the desired quantization direction, for example, to improve computational efficiency, reduce model complexity, or reduce model storage space. The user can select the desired quantization direction for all sub-models or for each sub-model individually. Figure 2C What is displayed is an interface for the user to select the desired quantization direction for all sub-models as a whole.

[0085] After the server obtains the divided sub-models and quantization directions, it can select the corresponding quantization algorithm and strategy for each sub-model based on the quantization direction. After completion, the client interface prompts the user to download the quantized deep learning model file. The user selects the download path on the interface and selects "Download" to download the quantized deep learning model file to the specified local path.

[0086] Quantification

[0087] Quantization refers to converting parameters such as weights in a deep learning model, as well as input tensors to the model, from high-precision data types to low-precision data types. This is explained below using a convolutional neural network as an example. A convolutional neural network consists of multiple layers, each with multiple nodes. A node is the smallest computational unit in a deep learning model that performs operations on input to produce output features. The features output by a node in each layer are transmitted to the nodes in the next layer, serving as input for the next layer. The convolutional layer performs matrix multiplication of the convolution kernel and adds an offset matrix to produce a feature map. The convolution kernel is a two-dimensional matrix, and each value in the matrix can be considered a weight parameter. Since convolutional neural networks have multiple layers, and each layer's convolution kernel contains multiple weight parameters, the number of weight parameters is enormous. Correspondingly, the number of input tensors in the input matrix multiplied by the convolution kernel is also enormous. Initial deep learning models often use high-precision models, where node weights and input tensors are high-precision numbers, such as 32-bit floating-point numbers. However, due to throughput and storage space limitations in actual deployments, this can be very inefficient. This requires converting parameters such as weights and input tensors for each node in the deep learning model from high-precision values to low-precision values. This is known as quantization, for example, converting 32-bit single-precision floating-point numbers (FP32) to 8-bit integers (INT8). While this reduces precision, it helps reduce data throughput and storage requirements. Only after quantization is complete can the quantized deep learning model be deployed locally or in the cloud.

[0088] Quantization can be performed in steps. For example, calibration is performed first to obtain a calibrated model, and then quantization is performed on the calibrated model to obtain a quantized deep learning model. Calibration is the act of determining the value range of the input tensor for each node in the deep learning model to be quantized. The input tensors of a node in the deep learning model to be quantized can be extracted at several time points, and the value range of the input tensor for that node can be summarized from the input tensors at these time points. Calibration is the foundation of quantization.

[0089] In some cases, calibration and quantization are performed together. This means that the input is the deep learning model to be quantized, and the output is the quantized deep learning model after calibration and quantization. After the quantization operation is completed, the high-precision model is converted to a low-precision model.

[0090] Quantization of deep learning models in the disclosed embodiments

[0091] In existing deployment frameworks, a single quantization algorithm and strategy is typically used for the entire deep learning model. A quantization algorithm refers to the algorithm used during quantization, such as symmetric quantization 511 and asymmetric quantization 512. A quantization strategy refers to the principles that must be followed during the quantization process in addition to the quantization algorithm, including the calibration algorithm and the number of bits to be quantized (the number of bits to which parameters such as the model's input tensors and weights are quantized after quantization). Calibration algorithms include the KL divergence algorithm 521, the Min-Max algorithm 522, and the moving average algorithm 523.

[0092] For example, in the quantization of the Mobilenet V1 model, existing techniques might select an asymmetric quantization algorithm and a KL divergence calibration algorithm for the entire model, and then uniformly apply these algorithms and strategies to the quantization of the entire model. This approach either results in a significant loss of time for quantization calibration or degrades the quantization accuracy of some models.

[0093] In the embodiment of the present disclosure, the deep learning model is divided into sub-models. Different sub-models may require different quantization algorithms and applicable quantization strategies. The embodiment of the present disclosure no longer limits a single model to a single quantization algorithm and strategy, but divides the sub-models so that different sub-models adopt different quantization algorithms and strategies, which not only prevents the huge time loss caused by quantization calibration, but also prevents the explosion of quantization accuracy. The embodiment of the present disclosure proposes a quantization scheme based on the fusion of quantization algorithms and strategies and the perception of sub-models. This scheme can comprehensively consider the characteristics of each sub-model in the model through learning, and then assign different quantization algorithms and strategies to each sub-model, so that the final quantization result can not only ensure quantization accuracy, but also reduce the complexity of quantization calibration.

[0094] The following describes in detail the process of quantizing and deploying the deep learning model in accordance with the embodiments of the present disclosure.

[0095] According to an embodiment of the present disclosure, a quantization method for a deep learning model is provided. Figure 1 All servers are jointly executed. Figure 4 As shown, the method includes:

[0096] Step 610: Receive a deep learning model to be quantized;

[0097] Step 620: Divide the deep learning model to be quantized into sub-models;

[0098] Step 630: For the sub-model, select a quantization algorithm and a quantization strategy corresponding to the sub-model from a combination of preset candidate quantization algorithms and candidate quantization strategies, where the quantization strategy is a criterion to be followed in addition to the quantization algorithm during the quantization process.

[0099] Step 640: Output the sub-model according to the corresponding quantization algorithm and quantization strategy to obtain a quantized deep learning model.

[0100] The above steps are described in detail below.

[0101] The receiving of the deep learning model to be quantized in step 610 may be performed by the above-mentioned entry server. Figure 2A As shown, an interface is displayed that guides the user to upload files of the deep learning model.

[0102] The sub-model mentioned in step 620 is part of the deep learning model. The nodes contained in this part use the same quantization algorithm and quantization strategy during quantization. Step 620 can be implemented using a sub-model partitioning model. That is, the deep learning model is input into the sub-model partitioning model, and the sub-model partitioning model outputs the partitioned sub-models.

[0103] The sub-model partitioning model is a neural network model specifically used in the embodiments of the present disclosure to partition a deep learning model into sub-models. It can be trained by inputting a deep learning model sample set consisting of a large number of deep learning model samples into the neural network model, which then outputs its partitioned sub-models. Steps 630 and 640 are then executed for these sub-models, and the resulting quantized deep learning model is deployed and executed to obtain the corresponding accuracy. If the number of samples in the deep learning model sample set with an accuracy greater than a predetermined accuracy threshold exceeds a predetermined ratio (e.g., 95%) of the total number of samples in the sample set, the model is considered successfully trained. Otherwise, the coefficients in the neural network model are adjusted so that the number of samples in the deep learning model sample set with an accuracy greater than a predetermined accuracy threshold exceeds a predetermined ratio (e.g., 95%) of the total number of samples in the sample set. A successfully trained neural network model can be used as a sub-model partitioning model. Any deep learning model can be input into the sub-model partitioning model to obtain the partitioned sub-models. The advantage of using a sub-model partitioning model is that it improves the automation and accuracy of sub-model partitioning.

[0104] In step 620, the nodes in the deep learning model may also be divided into sub-models according to predetermined rules based on their type and level. The node type refers to the type of operation performed by the node. For example, a node that performs a convolution operation is a convolution node. The node level refers to the layer of the deep learning model. As described above, deep learning models are hierarchical structures, and each layer may contain multiple nodes. The output of the nodes in the previous layer serves as the input of the nodes in the next layer.

[0105] Predefined rules are pre-defined rules for partitioning submodels based on node type and hierarchy. For example, a predefined rule might specify that all nodes of the same type in a deep learning model are grouped into a single submodel. Thus, convolution nodes in a deep learning model are grouped into a single submodel, pooling nodes into a single submodel, and so on. Alternatively, a predefined rule might specify that each node in a layer of a deep learning model is a submodel. Thus, regardless of node type, all nodes in the first layer are grouped into a single submodel, all nodes in the second layer into a single submodel, and so on. Rules can also be formulated based on a combination of type and hierarchy. For example, a predefined rule might specify that nodes in each layer of a deep learning model are grouped into different submodels based on type. Thus, convolution nodes in the first layer are grouped into a single submodel, pooling nodes in the first layer are grouped into a single submodel, convolution nodes in the second layer are grouped into a single submodel, pooling nodes in the second layer are grouped into a single submodel, and so on. Grouping based on predefined rules results in faster processing and consumes less processing resources.

[0106] In addition, you can also Figure 2B As shown, the user's manual division of the sub-model is accepted to improve the flexibility of dividing the sub-model.

[0107] At this time, after the entry server receives the deep learning model file to be quantized, it converts the deep learning model file to be quantized into a corresponding static graph through existing methods and displays it on the client. The static graph is a graph that represents the nodes in the deep learning model and the relationship between them. It shows the nodes of each layer and uses arrows to represent the signal input and output relationship between the nodes. Figure 3 As shown, the original model 501 (the deep learning model to be quantized) is represented by its static graph 504. Static graph 504 has four layers, with four nodes in each layer. Arrows between nodes in the previous layer and nodes in the next layer represent the input and output relationships of data. Arrows lead to output nodes, and arrows point to input nodes.

[0108] Users can Figure 2B The sub-model division instruction is made in the static graph shown. The sub-model division instruction is an instruction made by the user on the static graph to divide certain nodes into a sub-model. For example, the user draws a circle in the static graph, and the circled nodes are divided into a sub-model. The touch display receives the user's sub-model division instruction for the static graph and divides the deep learning model into sub-models according to the sub-model division instruction. Figure 3 As shown, the nodes of the 1st-2nd layer are divided into a sub-model, the nodes of the 2nd-3rd layer are divided into a sub-model, and the nodes of the 3rd-4th layer are divided into a sub-model.

[0109] Next, in step 630 , for the sub-model, a quantization algorithm and a quantization strategy corresponding to the sub-model are selected from a combination of preset candidate quantization algorithms and candidate quantization strategies.

[0110] The combination of candidate quantization algorithms and candidate quantization strategies is pre-set. Quantization algorithms compatible with the acceleration unit 230 are obtained as candidate quantization algorithms. Candidate quantization algorithms are candidate quantization algorithms from which the quantization algorithms used for quantization are selected. Not all quantization algorithms can be accelerated on the acceleration unit 230. Therefore, the quantization algorithms that can be accelerated by the acceleration unit 230 can be listed as candidate quantization algorithms. Quantization strategies compatible with the acceleration unit 230 are obtained as candidate quantization strategies. Candidate quantization strategies are candidate quantization strategies from which the quantization strategies used for quantization are selected. Not all quantization strategies can be accelerated on the acceleration unit 230. Therefore, the quantization strategies that can be accelerated by the acceleration unit 230 can be listed as candidate quantization strategies. Next, the listed candidate quantization algorithms and the listed candidate quantization strategies are combined with each other. For example, there are two candidate quantization algorithms: the symmetric quantization algorithm sys and the asymmetric quantization algorithm asym. Since quantization often requires a uniform number of quantization bits (for example, when quantizing a 32-bit floating-point number to an 8-bit integer, the number of quantization bits is uniformly 8), in most cases, the number of quantization bits cannot be selected (although multiple options are possible in some cases). Candidate quantization strategies are mainly targeted at calibration algorithms, and candidate quantization strategies include KL divergence, Min-Max, and moving average. Thus, the combination of candidate quantization algorithms and candidate quantization strategies A = {sys, asys} * {KL divergence, Min-Max, moving average} ultimately produces six combinations: sys with KL divergence, sys with Min-Max, sys with moving average, asys with KL divergence, asys with Min-Max, and asys with moving average.

[0111] From a pre-set combination of candidate quantization algorithms and candidate quantization strategies, a quantization algorithm and quantization strategy corresponding to the sub-model can be selected through a pre-trained quantization algorithm and strategy selection model. The sub-model is input into the quantization algorithm and strategy selection model, and the quantization algorithm and strategy selection model outputs the selected quantization algorithm and quantization strategy.

[0112] The quantization algorithm and strategy selection model is a neural network model specifically configured to output the quantization algorithm and quantization strategy selected for a sub-model in an embodiment of the present disclosure, such as an evolutionary algorithm model or an xgboost model. It can be trained in the following manner: a deep learning model sample set consisting of a large number of deep learning model samples is divided into sub-models according to the same method in step 610, each sub-model is input into the neural network model, and the neural network model outputs the corresponding quantization algorithm and quantization strategy. In step 630, the deep learning model samples obtained after quantization of the sub-models according to the corresponding quantization algorithm and quantization strategy are deployed to the acceleration unit 230, which runs the model and obtains the accuracy corresponding to the deep learning model samples. If the number of samples in the deep learning model sample set with an accuracy greater than a predetermined accuracy threshold exceeds a predetermined ratio (e.g., 95%) of the total number of samples in the sample set, the model is considered to have been trained successfully. Otherwise, the coefficients in the neural network model are adjusted so that the number of samples in the deep learning model sample set with an accuracy greater than a predetermined accuracy threshold exceeds a predetermined ratio (e.g., 95%) of the total number of samples in the sample set. The successfully trained neural network model can then be used as the quantization algorithm and strategy selection model. By inputting any sub-model into the quantitative algorithm and strategy selection model, the selected quantitative algorithm and quantitative strategy can be obtained. The advantage of using the quantitative algorithm and strategy selection model to select quantitative algorithms and quantitative strategies is that it can improve the accuracy of the selected quantitative algorithms and strategies.

[0113] In the above embodiment, the quantization algorithm and strategy are selected only based on the sub-model. In another embodiment, the quantization algorithm and strategy can also be selected in combination with the quantization direction input by the user. In this embodiment, before step 630, the method further includes: receiving the quantization direction input by the user. The quantization direction refers to the goal pursued by quantization. For example, there are different quantization directions such as improving computational efficiency, reducing model complexity, and reducing model storage space. Different quantization directions determine the selection of quantization algorithms and quantization strategies. It can be as follows: Figure 2C As shown, this is achieved by listing all candidate quantization directions and allowing users to select them on the interface.

[0114] In this embodiment, a quantization algorithm and strategy must be selected based on the sub-model and the quantization direction corresponding to the sub-model. This can be done by pre-training a quantization algorithm and strategy selection model. The sub-model and quantization direction are input into the quantization algorithm and strategy selection model, which then outputs the selected quantization algorithm and strategy.

[0115] In this case, the quantization algorithm and strategy selection model can be trained in the following manner: for a sample pair set consisting of a large number of sample pairs of deep learning models and quantization directions, the deep learning model in each sample pair is divided into sub-models according to the same method as in step 620. These sub-models are then input into a neural network model along with the quantization direction in the sample pair, and the neural network model outputs the corresponding quantization algorithm and quantization strategy. The deep learning model obtained by quantizing the sub-model according to the corresponding quantization algorithm and quantization strategy is deployed and executed to obtain the accuracy corresponding to the quantized deep learning model sample. If the number of sample pairs in the sample pair set with an accuracy greater than a predetermined accuracy threshold exceeds a predetermined ratio (e.g., 95%) of the total number of samples in the sample pair set, the model training is considered successful. Otherwise, the coefficients in the neural network model are adjusted so that the number of samples in the sample pair set with an accuracy greater than a predetermined accuracy threshold exceeds a predetermined ratio (e.g., 95%) of the total number of samples in the sample pair set. The successfully trained neural network model can be used as the quantization algorithm and strategy selection model. By inputting any sub-model and quantization direction into the quantization algorithm and strategy selection model, the selected quantization algorithm and quantization strategy can be obtained. The advantage of using the quantitative algorithm and strategy selection model to select quantitative algorithms and quantitative strategies is that it can improve the automation and accuracy of selecting quantitative algorithms and strategies.

[0116] In the above embodiment, the quantization direction is applied to all sub-models of the deep learning model to be quantized. In another embodiment, the quantization direction is applied to a single sub-model. That is, for each sub-model, the user input corresponding to the quantization direction of the sub-model is received. When selecting the quantization algorithm and quantization strategy, the sub-model and the quantization direction corresponding to the sub-model are also input into the quantization algorithm and strategy selection model to obtain the selected quantization algorithm and quantization strategy.

[0117] In addition, users can also manually select quantitative algorithms and quantitative strategies to increase the flexibility of determining quantitative algorithms and quantitative strategies. In this way, quantitative algorithms and strategies can be customized and scalable.

[0118] At this time, a combination of pre-set candidate quantization algorithms and candidate quantization strategies (not shown) can be displayed on the client. The user selects the desired quantization algorithm and quantization strategy from the displayed combination. The server quantizes according to the quantization algorithm and quantization strategy selected by the user.

[0119] As described above, when a deep learning model performs inference, a node multiplies the input tensor by its own weight and adds the offset corresponding to the node to obtain an output tensor that is output to the node in the next layer. Therefore, not only the weights need to be quantized, but the input tensors and offsets also need to be quantized. In one embodiment, the same quantization algorithm and strategy can be used for the input tensors, weights, and offsets of the nodes. In another embodiment, corresponding quantization algorithms and quantization strategies can be selected for the input tensors, weights, and offsets of the nodes in the sub-model. For example, when the quantization algorithm and strategy selection model selects the quantization algorithm and strategy, the model can select different quantization algorithms and quantization strategies for the input tensors, weights, and offsets of the nodes respectively. For example, in some cases, when quantizing a 32-bit floating-point number to an 8-bit integer, it is sufficient to only set the number of bits of the input tensor and weights after quantization to 8, and it is not necessary to set the number of bits of the offset to 8 as well. The number of bits of the offset after quantization can be set to 32.

[0120] Then, in step 640, the output sub-model is quantized according to the corresponding quantization algorithm and quantization strategy to obtain a quantized deep learning model.

[0121] Quantization involves a process called calibration and quantization based on the maximum and minimum values obtained from calibration. Quantization first involves determining the value range of the input tensor for each node in the model. The maximum value of the range is then aligned with the quantized maximum value, and the minimum value of the range is aligned with the quantized minimum value. Determining the value range of the input tensor for each node in a deep learning model is called calibration. The input tensors of a node in the deep learning model at several time points can be extracted and the value range of the node's input tensor can be summarized from the input tensors at these time points.

[0122] Furthermore, because deep learning models are written in different frameworks, such as TensorFlow, MXNet, and Caffe, quantization for these different frameworks or formats requires separate quantization methods adapted to each format. To unify subsequent quantization processes, a conversion process is implemented: This involves converting the formats of various frameworks, such as TensorFlow, MXNet, and Caffe, into a common intermediate representation (IR) that is independent of any particular framework or format. Quantization is performed on this common intermediate representation, and then converted back to the original framework or format, achieving unified quantization across different frameworks.

[0123] Therefore, in the embodiment of the present disclosure, when quantizing a sub-model according to the corresponding quantization algorithm and quantization strategy, the model can first be converted into a universal intermediate representation (IR), and a maximum and minimum value collection node for collecting the maximum and minimum values of the input to be quantized can be inserted into the IR. The maximum and minimum value collection node is a small program inserted, which is used to perform calibration operations according to the calibration algorithm during execution. Then, the universal intermediate representation with the maximum and minimum value collection node inserted is converted back to the specific format before conversion (such as TensorFlow 408, etc.) for calibration to obtain the maximum and minimum values of the input to be quantized. At this time, the deep learning model becomes a calibration model. Then, based on the obtained maximum and minimum values of the input to be quantized, a quantization node is inserted into the universal intermediate representation. The quantization node is a small program inserted, which is used to perform quantization operations according to the quantization algorithm and the number of bits after quantization during execution. The universal intermediate representation with the quantization node inserted is converted back to the specific format before conversion, quantized, and a quantized deep learning model is obtained, which can then be deployed.

[0124] like Figure 2D As shown, the quantized deep learning model obtained by quantizing the output sub-model according to the corresponding quantization algorithm and quantization strategy can be downloaded by prompting the user on the client interface. The user specifies the local download path of the quantized deep learning model file and selects Download. The quantized deep learning model file can be downloaded to the specified local download path.

[0125] The embodiments of the present disclosure Figure 4 When the above quantization method is executed in the cloud, the different steps of the method can be performed by Figure 1 The codes of different steps are stored on different servers 140 and are executed by the processors on the servers 140 to achieve the following: Figure 4 In addition, the embodiment of the present disclosure is as follows Figure 4 The above quantization method shown can also be executed on a single machine instead of on the cloud. Figure 4 The above quantization method is performed by a computer device. The computer device includes a memory and a processor. The memory is used to store Figure 4 The processor is used to execute the computer executable code stored in the memory to implement the quantization method shown. Figure 4 The quantification method shown.

[0126] The commercial value of the disclosed embodiments

[0127] The disclosed embodiment proposes a quantization scheme based on the fusion of quantization algorithms and strategies and the perception of sub-models. This scheme can comprehensively consider the characteristics of each sub-model in the model through learning, and then assign different quantization algorithms and strategies to each sub-model, so that the final quantization result can both ensure quantization accuracy and reduce the complexity of quantization calibration. Experiments have shown that the final quantization accuracy is increased to 1.5 times the original, and the complexity is only less than 80% of the original. It has good market prospects in optimizing the deployment and execution of deep learning models.

[0128] It should be understood that the various embodiments in this specification are described in a progressive manner. References to the same or similar parts of the various embodiments will be sufficient. Each embodiment focuses on the differences from the other embodiments. In particular, the device embodiments are described briefly because they are generally similar to the process of the method embodiments. For relevant details, references to the description of the method embodiments will suffice.

[0129] It should be understood that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0130] It should be understood that an element described herein in the singular or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown herein as separate may be combined into a single module or element, and modules or elements described or shown herein as single may be split into multiple modules or elements.

[0131] It should also be understood that the terms and expressions used herein are for descriptive purposes only, and the one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or portions thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A quantization method for a deep learning model, comprising: Receiving a deep learning model to be quantized, wherein the deep learning model is used to realize recognition of at least one type of data selected from text, image, and sound; Dividing the deep learning model to be quantized into sub-models includes: generating and displaying a static image corresponding to the deep learning model to be quantized; receiving a user's instruction on dividing the static image into sub-models by touching the display; and dividing the deep learning model into sub-models according to the instruction on dividing the sub-models; Obtain a quantization algorithm compatible with the acceleration unit as a candidate quantization algorithm; obtain a quantization strategy compatible with the acceleration unit as a candidate quantization strategy; Receive the quantitative direction input by the user on the interface; For the submodel, selecting a quantization algorithm and a quantization strategy corresponding to the submodel from a combination of preset candidate quantization algorithms and candidate quantization strategies, including: selecting a quantization algorithm and a quantization strategy corresponding to the submodel and the quantization direction, the quantization strategy being a criterion to be followed in addition to the quantization algorithm during the quantization process, the quantization direction including at least one of the following: improving computational efficiency, reducing model complexity, and reducing model storage space; The output sub-model is quantized according to the corresponding quantization algorithm and quantization strategy to obtain the quantized deep learning model.

2. The method according to claim 1, wherein The receiving of the quantization direction input by the user includes: receiving the quantization direction input by the user corresponding to the divided sub-model; The selecting of the quantization algorithm and the quantization strategy corresponding to the sub-model and the quantization direction includes: selecting the quantization algorithm and the quantization strategy according to the sub-model and the quantization direction corresponding to the divided sub-models.

3. The method according to claim 1, wherein The step of dividing the deep learning model to be quantized into sub-models includes: The deep learning model to be quantized is input into a sub-model partitioning model, and the sub-model partitioning model outputs the partitioned sub-model.

4. The method according to claim 1, wherein The step of dividing the deep learning model to be quantized into sub-models further includes: Based on at least one of the type and level of the nodes in the deep learning model to be quantized, the nodes are divided into sub-models according to predetermined rules.

5. The method according to claim 1, wherein The quantization strategy includes at least one of a calibration algorithm and a number of quantized bits.

6. The method according to claim 1, wherein The combination of the candidate quantization algorithm and the candidate quantization strategy is preset in the following manner: Acquire a quantization algorithm compatible with the acceleration unit as a candidate quantization algorithm; Acquire a quantization strategy compatible with the acceleration unit as a candidate quantization strategy; The candidate quantization algorithms and the candidate quantization strategies are combined with each other.

7. The method according to claim 1, wherein The selecting of a quantization algorithm and a quantization strategy corresponding to the sub-model and the quantization direction includes: The sub-model and the quantization direction are input into a quantization algorithm and strategy selection model, and the quantization algorithm and strategy selection model outputs the selected quantization algorithm and quantization strategy.

8. The method according to claim 1, wherein The selecting of a quantization algorithm and a quantization strategy corresponding to the sub-model includes: Displaying the combination of the preset candidate quantization algorithm and candidate quantization strategy; receiving a user's selection indication from the combination of the preset candidate quantization algorithms and candidate quantization strategies; According to the selection instruction, a quantization algorithm and a quantization strategy corresponding to the sub-model are selected.

9. The method according to claim 1, wherein The selecting of a quantization algorithm and a quantization strategy corresponding to the sub-model includes: For the input tensors, weights, and offsets of the nodes in this sub-model, select the corresponding quantization algorithm and quantization strategy respectively.

10. The method according to claim 5, wherein Quantize the sub-model according to the corresponding quantization algorithm and quantization strategy, including: calibrating the nodes of the sub-model according to the calibration algorithm; The calibrated nodes are quantized according to the quantization algorithm and the quantized bit number.

11. A cloud service system comprising a plurality of cloud servers for distributedly storing computer executable codes, wherein when the computer executable codes are executed by processors on corresponding cloud servers, the method according to any one of claims 1 to 10 is implemented.

12. A data center comprising the cloud service system according to claim 11.

13. A computer device comprising: a memory for storing computer executable code; A processor, configured to execute the computer executable code stored in the memory to implement the method according to any one of claims 1-10.

Images