Quantitative method, device, equipment and medium for distributed deployment model

Abstract

The present disclosure provides a quantization method, device, equipment and medium for a distributed deployment model, which relates to the field of artificial intelligence, and specifically to technical fields such as deep learning, distributed training and model quantization. The specific implementation scheme of the quantization method for the distributed deployment model is: for multiple network layers included in the deployed model, determine the loss value of each network layer in the multiple network layers according to the first quantization accuracy as the quantization loss value of each network layer; send the quantization loss values ​​of the multiple network layers to the target node in the multiple computing nodes of the distributed deployment model, so that the target node determines the target network layer in the multiple network layers according to the received quantization loss value; and quantize the network parameters of the target network layer according to the first quantization accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to technical fields such as deep learning, distributed training and model quantization, and in particular to a quantization method, device, equipment and medium for a distributed deployment model. Background Art

[0002] With the development of computer technology and network technology, the application of neural network models is becoming more and more extensive. At the same time, higher and higher requirements are placed on the accuracy of neural network models. In order to meet the accuracy requirements, neural network models usually have a large number of parameters, and the demand for storage space and computing resources is large, which brings huge challenges to the deployment and application of neural network models.

[0003] In order to meet this huge challenge, a variety of model compression methods have emerged. However, model compression will inevitably lead to a loss of model accuracy. Summary of the invention

[0004] The present disclosure aims to provide a quantization method, apparatus, device and medium for a distributed deployment model to improve the accuracy of the quantized model.

[0005] According to a first aspect of the present disclosure, a quantization method for a distributed deployment model is provided, comprising: for multiple network layers included in the deployed model, determining a loss value for each of the multiple network layers according to a first quantization accuracy as a quantization loss value for each network layer; sending the quantization loss values ​​of the multiple network layers to a target node among multiple computing nodes of the distributed deployment model, so that the target node determines a target network layer among the multiple network layers according to the received quantization loss value; and quantizing network parameters of the target network layer according to the first quantization accuracy.

[0006] According to a second aspect of the present disclosure, a quantization device for a distributed deployment model is provided, including: a first loss determination module, for determining, for a plurality of network layers included in the deployed model, a loss value of each of the plurality of network layers quantized according to a first quantization accuracy, as a quantization loss value of each network layer; a first loss sending module, for sending the quantization loss values ​​of the plurality of network layers to a target node among a plurality of computing nodes of the distributed deployment model, so that the target node determines a target network layer among the plurality of network layers according to the received quantization loss value; and a parameter quantization module, for quantizing a network parameter of the target network layer according to the first quantization accuracy.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the quantization method of the distributed deployment model provided by the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the quantization method of the distributed deployment model provided by the present disclosure.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program / instruction, when executed by a processor, implements the quantization method of the distributed deployment model provided by the present disclosure.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 It is a schematic diagram of an application scenario of a quantization method and device for a distributed deployment model according to an embodiment of the present disclosure;

[0013] Figure 2 is a flow chart of a quantization method of a distributed deployment model according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of the principle of determining the loss value of each network layer quantized according to the first quantization accuracy according to an embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram of an implementation principle of a quantization method for a distributed deployment model according to an embodiment of the present disclosure;

[0016] Figure 5 is a schematic diagram of the principle of determining a scaling factor of an input activation value of a self-attention layer according to an embodiment of the present disclosure;

[0017] Figure 6 is a schematic diagram of an application of a quantization method for a distributed deployment model according to an embodiment of the present disclosure;

[0018] Figure 7is a structural block diagram of a quantization device of a distributed deployment model according to an embodiment of the present disclosure; and

[0019] Figure 8 It is a block diagram of an electronic device for implementing the quantization method of the distributed deployment model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] With the development of deep learning technology, the number of parameters in neural network models is getting larger and larger. For example, the maximum number of parameters in the Generative Pre-Trained Transformer model can reach hundreds of millions. The deployment of such a large-scale model is a huge challenge. Generally, a GPU server with Multicard A100 is required to perform distributed deployment and application of such a large-scale model.

[0022] In order to speed up the deployment of these large-scale models, various compression model methods have emerged. Among the various compression model methods, the most widely used model quantization methods can be divided into quantization-aware training (Quantization Aware Training, referred to as QAT) method and post-training quantization (Post Training Quantization, referred to as PTQ) method. Among them, the QAT method requires fine-tuning training of the full-precision model, which is time-consuming and has a long cycle. The PTQ method is also called an offline quantization method. This method runs the model forward reasoning, and needs to collect the truncation value range of the layer to be quantized, and convert the corresponding operations in the model into low-bit operations according to this truncation value. At the same time, the network parameters of the model will also be converted into low-bit parameters. The PTQ method can quickly produce a quantized model, and since the model does not need to be trained, only a small number of samples are required as calibration data to participate in the quantization process. However, the PTQ method will result in a loss of model accuracy.

[0023] In order to solve the above problems, the present disclosure provides a quantification method, device, equipment and medium for a distributed deployment model. Figure 1 The application scenarios of the method and device provided by the present disclosure are described.

[0024] Figure 1It is a schematic diagram of an application scenario of a quantization method and device for a distributed deployment model according to an embodiment of the present disclosure.

[0025] like Figure 1 As shown, the application scenario 100 of this embodiment may include a distributed system 110, and the distributed system 110 has a plurality of computing nodes 111-113, and the plurality of computing nodes 111-113 are distributedly deployed with a neural network model 120 including a plurality of network layers. Specifically, each network layer of the neural network model 120 may be divided into a plurality of network blocks, and the plurality of network blocks may be deployed one-to-one on a plurality of computing nodes 111-113, and the plurality of computing nodes 111-113 may train the network blocks deployed respectively to obtain a trained neural network model.

[0026] In one embodiment, the application scenario 100 may further include a master node 130, which may be any one of the multiple computing nodes set in the distributed system 110, or a node independent of the distributed system 110. This embodiment may determine the quantization configuration information through communication interaction between the master node 130 and the multiple computing nodes 111 to 113, and the multiple computing nodes 111 to 113 may quantize the network blocks deployed respectively according to the quantization configuration information to achieve quantization of the trained neural network model. This embodiment may obtain the prediction model 140 by deriving the quantized network blocks.

[0027] The network parameters of the prediction model 140 significantly reduce the requirements for computing resources and storage space compared to the neural network model 120. For example, the prediction model 140 can be deployed in an electronic device 150 with limited computing power, so that the electronic device 150 uses the deployed prediction model 140 to process data generated online.

[0028] For example, the neural network model 120 may be a text processing model, an image processing model, or a speech processing model, etc. Specifically, the neural network model 120 may be a generative model such as a generative language model, an image generation model, or a speech generation model, which is not limited in the present disclosure.

[0029] It should be noted that the quantization method of the distributed deployment model provided in the present disclosure can be executed by any computing node in the distributed system 110. Accordingly, the quantization device of the distributed deployment model provided in the present disclosure can be set in any computing node in the distributed system 110.

[0030] It should be understood that Figure 1 The number and type of computing nodes included in the distributed system 110 are merely illustrative. According to implementation requirements, the distributed system 110 may have any number and type of computing nodes.

[0031] The following will be combined Figure 2 to Figure 6 The quantification method of the distributed deployment model provided by the present disclosure is described in detail.

[0032] Figure 2 It is a flowchart of a quantization method of a distributed deployment model according to an embodiment of the present disclosure.

[0033] like Figure 2 As shown, the quantization method 200 of the distributed deployment model of this embodiment may include operations S210 to S230. The method 200 may be executed by each computing node of the aforementioned multiple computing nodes, for example.

[0034] In operation S210, for a plurality of network layers included in the deployed model, a loss value of each of the plurality of network layers quantized according to a first quantization accuracy is determined as a quantization loss value of each network layer.

[0035] According to an embodiment of the present disclosure, the first quantization precision may be, for example, a quantization precision corresponding to int8, or a quantization precision corresponding to a format such as int4.

[0036] This embodiment can determine the upper limit Q of the value of the network parameter in the network layer after quantization according to the first quantization accuracy. max And the lower limit Q min For example, for the quantization accuracy corresponding to int8, the upper limit Q is determined max Can be 127, the lower limit is Q min It can be -128. At the same time, the upper limit T of the network parameters in each network layer can be determined. max And the lower limit T min Then, we can use the upper limit Q max , the lower limit Q min , the upper limit T of the network parameters in each network layer max And the lower limit T min , determine the scaling factor S of the network parameters of each network layer. Subsequently, each network layer is pre-quantized according to the scaling factor, that is, the network parameters of each network layer are converted into parameters in a format such as int8, and a pre-quantized network layer corresponding to each network layer is obtained. It can be understood that a symmetric quantization method or an asymmetric quantization method can be used for pre-quantization.

[0037] For example, the scaling factor S can be calculated using the following formula (1).

[0038]

[0039] For example, if an asymmetric quantization method is used, the following formulas (2) to (4) can be used to pre-quantize the network parameters of each network layer.

[0040]

[0041]

[0042]

[0043] Among them, Q represents the fixed-point number after quantization, T represents the network parameters, and Z represents the offset of the network parameters of each network layer, that is, the fixed-point value corresponding to the floating-point number 0 after the network parameters in the form of floating-point numbers are mapped to fixed-point numbers. S is the scaling factor, that is, the proportional relationship between the floating-point number and the fixed-point number. The Round() function represents the rounding operation, and the clamp() function is used to limit the value between the upper limit and the lower limit. It can be understood that when the symmetric quantization method is used for calculation, the value of the above Z value is 0.

[0044] Subsequently, the embodiment may use the pre-quantized network layer to replace each network layer in the model deployed by the computing node to obtain a replaced model. The model deployed by the computing node and the replaced model may be used to process the same predetermined calibration data to obtain two processing results. The embodiment may determine the loss value of quantizing each network layer according to the first quantization accuracy based on the difference between the two processing results. For example, the difference between the two processing results may be positively correlated with the loss value. For example, for a text generation model, the processing result may be a generated predicted text; for an image generation model, the processing result may be a generated predicted image.

[0045] For example, the predetermined calibration data may have a label of the actual processing result, and the embodiment may compare each of the two processing results with the actual processing result to obtain two comparison results. For example, a cross entropy loss function may be used to calculate the comparison result between each processing result and the actual processing result. Subsequently, the difference between the two comparison results is used to represent the loss value of each network layer.

[0046] In operation S220, the quantized loss values ​​of the multiple network layers are sent to a target node among the multiple computing nodes of the distributed deployment model, so that the target node determines a target network layer among the multiple network layers according to the received quantized loss values.

[0047] It is understandable that the target node can be, for example, a master node selected by voting of multiple computing nodes, or can be any one of the multiple computing nodes. Assuming that the neural network model includes M network layers, each computing node can obtain M quantized loss values ​​of M network layers through operation S210, and send the M quantized loss values ​​to the target node. In order to facilitate the distinction between the quantized loss values ​​of different network layers, each quantized loss value sent to the target node can carry the identification information of each network layer.

[0048] In this way, if the number of multiple computing nodes is P, the target node can receive P*M quantization loss values. The P*M quantization loss values ​​include P quantization loss values ​​for each network layer. The target node can, for example, take the average value or maximum value of the P quantization loss values ​​of each network layer as the target quantization loss value of each network layer. Subsequently, the target node can, for example, select a predetermined proportion of network layers with larger target quantization loss values ​​from multiple network layers as the target network layer. Alternatively, the target node can compare the target quantization loss value of each network layer with the predetermined loss value. A network layer with a target quantization loss value less than the predetermined loss value is selected from multiple network layers as the target network layer. Among them, the predetermined proportion and the predetermined loss value can be set according to actual needs, and the present disclosure does not limit this.

[0049] After the target node determines the target network layer, it uses the target network layer as the network layer that needs to be quantized, and sends indication information indicating the target network layer to the multiple computing nodes, so that the multiple computing nodes perform operation S230.

[0050] In operation S230, a network parameter of the target network layer is quantized according to a first quantization accuracy.

[0051] It is understandable that the process of quantizing the network parameters of the target network layer is similar to the principle of pre-quantizing the network layer described above. For example, if the value of the network parameter of the target network layer is a value in the float32 format, and the first quantization precision is the precision corresponding to the int8 format, the network parameter of the target network layer can be quantized by converting the network parameter of the target network layer from float32 to int8.

[0052] The method of the disclosed embodiment can guarantee a precision lossless quantization model to a large extent by performing a separate quantization analysis on each network layer in the neural network model and selecting a target quantization layer for quantization based on the result of the quantization analysis. Furthermore, by having each computing node perform a separate quantization analysis on each network layer in its respective deployed model, and having the target node summarize the quantization loss and determine the target network layer, the efficiency of the quantization analysis can be improved.

[0053] According to an embodiment of the present disclosure, for other network layers except the target network layer, for example, the quantization of these other network layers can be skipped, that is, these other network layers are not quantized, so as to improve the accuracy of the quantized model. Alternatively, the other network layers can be quantized according to a second quantization accuracy with higher accuracy, so as to minimize the accuracy loss of the quantized model compared to the pre-quantization model while ensuring that the quantized model has a smaller scale. For example, if the first quantization accuracy is the quantization accuracy corresponding to int8, the second quantization accuracy can be the quantization accuracy corresponding to int16, etc.

[0054] For example, for the other network layer, the second scaling factor for the network parameters of the other network layer can be determined according to the second quantization accuracy and the values ​​of the network parameters included in the other network layer. Subsequently, the network parameters of the other network layer are quantized according to the second scaling factor. Among them, the principle of determining the second scaling factor is similar to the principle of determining the scaling factor S described above, and the principle of quantizing the network parameters of the other network layer is similar to the principle of quantizing the network parameters of the target network layer in operation S230 described above, which will not be repeated here.

[0055] Figure 3 It is a schematic diagram of the principle of determining the loss value of each network layer quantized according to the first quantization accuracy according to an embodiment of the present disclosure.

[0056] According to an embodiment of the present disclosure, when determining the loss value of quantizing each network layer, for example, a symmetric quantization method can be used to pre-quantize the network parameters of each network layer, and a scaling factor can be determined based on the maximum value of the absolute values ​​of the network parameters included in each network layer. In this way, all network parameters in the network layer are retained to avoid the situation where the accuracy of the quantized model is impaired due to truncation of outliers. This is because in the process of realizing the concept of the present disclosure, the inventors found that in the neural network model, matrix multiplication operations involve a large number of outliers, which are very important for the prediction of the results. If the outliers are truncated, the accuracy of the quantized model will be greatly reduced.

[0057] like Figure 3 As shown, in this embodiment 300, for a certain network layer 310 included in the neural network model, each computing node can determine the absolute value 320 of the value of the network parameter in the certain network layer 310 deployed by it. Subsequently, the maximum value 321 of the determined absolute value 320 is determined, and the first scaling factor 340 of the network parameter for the certain network layer 310 is determined based on the maximum value 321 and the first quantization precision 330. For example, the method described above can be used to determine the upper limit Q of the value of the network parameter in the quantized network layer according to the first quantization precision 330. max And the lower limit Q min, and the determined maximum value 321 is set as abs_max(T), then the first scaling factor S1 can be determined by the following formula (5), or the first scaling factor S1 can be determined by the following formula (6). For example, if the first quantization precision is the quantization precision corresponding to int8, the value of b is 8.

[0058]

[0059]

[0060] After determining the first scaling factor 340, for example, the first scaling factor 340 may be used to quantize the network parameters of the certain network layer and determine the quantization loss value. For example, the following formula (7) may be used to quantize the network parameters of the certain network layer.

[0061]

[0062] For example, the embodiment may use the first scaling factor 340 to pre-quantize the network parameters of the certain network layer to obtain the pre-quantized network layer 350. Subsequently, the pre-quantized network layer 350 is used to replace the certain network layer 310 in the deployed model 360 to obtain the replaced model. Subsequently, the deployed model 360 and the replaced model are used to process the predetermined calibration data 370 in parallel to obtain two processing results. Subsequently, the loss value of quantizing the certain network layer 310 is determined based on the difference between the two processing results.

[0063] For example, after obtaining the prequantized network layer 350, the embodiment may also use the deployed model 360 to process the predetermined calibration data 370 to obtain the input activation value 380 for the certain network layer. Specifically, the predetermined calibration data may be input into the deployed model 360, and the activation value output by the previous network layer of the certain network layer in the deployed model 360 is used as the input activation value 380 of the certain network layer. Subsequently, the input activation value 380 is processed by the certain network layer 310 to obtain the first output activation value 391, and the input activation value 380 is processed by the prequantized network layer 350 to obtain the second output activation value 392. The loss value may be determined based on the distance between the first output activation value 391 and the second output activation value 392.

[0064] Specifically, the input activation value 380 can be input into a certain network layer 310 and the prequantized network layer 350 in parallel, and the output of the certain network layer 310 is used as the first output activation value 391, and the output of the prequantized network layer 350 is used as the second output activation value 392. When the prequantized network layer 350 is used to process the input activation value 380, for example, the input activation value 380 can be first quantized according to the first quantization accuracy, and the quantized activation value can be input into the prequantized network layer 350, so that the prequantized network layer 350 can calculate the input activation value according to the network parameters in the prequantized network layer 350. It can be understood that the principle of quantizing the input activation value 380 is similar to the principle of quantizing the network parameters in the network layer, and will not be repeated here.

[0065] For example, the distance between the first output activation value 391 and the second output activation value 392 can be used as the quantization loss value 3100 of each network layer. For example, the distance between two output activation values ​​can be Euclidean distance, cosine distance, Hamming distance, Manhattan distance, Jaccard distance, etc.

[0066] In one embodiment, multiple types of distances between the first output activation value 391 and the second output activation value 392 may be determined first, and then the loss value of quantizing the certain network layer 310 may be determined based on the multiple types of distances to obtain the quantized loss value 3100. For example, the Euclidean distance and the Earth Mover's Distance between the two output activation values ​​may be determined first. The sum of the two distances or the average of the two distances may then be used as the loss value of quantizing the certain network layer 310. The physical meaning of the Earth Mover's Distance is: changing a randomly distributed histogram into another randomly distributed histogram, and the minimum distance sum of the histogram changes.

[0067] This embodiment determines the quantization loss value based on the distance between the activation value output by the network layer before quantization and the activation value output by the network layer after quantization, so that the determined quantization loss value can better reflect the difference in processing power between the network layer before quantization and the network layer after quantization. Thereby, the accuracy of the determined quantization loss value can be improved, which is conducive to improving the accuracy of the selected target network layer. Furthermore, by combining multiple types of distances to determine the quantization loss value, the accuracy of the determined quantization loss value can be further improved.

[0068] The disclosed embodiment determines the scaling factor according to the maximum absolute value of the network parameter values, and determines the target network layer quantized according to the first quantization accuracy according to the aforementioned determined quantization loss value. This can compensate for the poor rationality of the scaling ratio setting and the large precision loss caused by quantization when there is a large proportion of outliers in the network parameter values, and thus can improve the accuracy of the quantized model.

[0069] In one embodiment, the number of predetermined calibration data may be R, and each computing node may obtain a loss value for each network layer by processing each predetermined calibration data, and a total of R loss values ​​may be obtained. In this embodiment, the sum of the R loss values ​​may be used as the quantization loss value of each network layer determined by each computing node. Accordingly, if the value of the predetermined loss value is L, when the target calculation determines whether each network layer is a target network layer, the target quantization loss value of each network layer determined may be compared with L*R. If the target quantization loss value is less than L*R, each network layer is determined to be a target network layer. Wherein, R is a natural number greater than 1. By setting multiple predetermined calibration data, the accuracy of the determined target network layer can be improved. This is because the processing accuracy of the network layer for different sample data (i.e., predetermined calibration data) is slightly different. By comprehensively considering multiple predetermined calibration data to determine the quantization loss value, the determined quantization loss value can better express the difference in accuracy of the network layer before and after quantization. Among them, L can be any value such as 0.06, and R can be any value such as 5, 10, etc. The values ​​of L and R can be set according to actual needs, and the present disclosure does not limit this.

[0070] Figure 4 It is a schematic diagram of the implementation principle of the quantization method of the distributed deployment model according to an embodiment of the present disclosure.

[0071] like Figure 4 As shown, in this embodiment 400 , the quantization method of the distributed deployment model is implemented by multiple computing nodes 410 and target nodes 420 deployed in the distributed system, wherein the target node 420 can be any one of the multiple computing nodes 410 .

[0072] In this embodiment 400 , each computing node 410 may first perform operations S411 to S412 .

[0073] In operation S411, each network layer in the deployed model is pre-quantized according to the first quantization precision to obtain a quantization loss value of each network layer. The implementation principle of operation S411 is similar to that of operation S210 described above, and will not be repeated here.

[0074] In operation S412 , quantized loss values ​​of a plurality of network layers are transmitted to the target node 420 .

[0075] After the target node 420 receives the quantized loss values ​​of multiple network layers sent by multiple computing nodes, operations S421 to S422 may be performed.

[0076] In operation S421, based on the received multiple quantization loss values ​​of each network layer, it is determined whether each network layer is a target network layer, so as to filter out the target network layer from the multiple network layers included in the neural network model. In operation S422, indication information indicating the target network layer is sent to multiple computing nodes 410. Among them, the target node can, for example, first determine the maximum value of the multiple quantization loss values ​​of each network layer, and then determine whether the maximum value is less than a predetermined loss value. If so, each network layer is determined to be a target quantization layer.

[0077] In response to receiving the indication information indicating the target network layer, the plurality of computing nodes 410 may perform operations S413 and S414. It is understandable that operations S413 and S414 may be performed in parallel or in any order, which is not limited in the present disclosure.

[0078] In operation S413, a network parameter of the target network layer is quantized according to a first quantization accuracy.

[0079] In operation S414, network parameters of other network layers among the plurality of network layers except the target network layer are quantized according to the second quantization accuracy.

[0080] For example, in this embodiment, the first quantization precision may be the quantization precision corresponding to int8, and the second quantization precision may be the quantization precision corresponding to int16. When quantizing or pre-quantizing the network parameters of the network layer according to the quantization precision, for example, the scaling factor may be determined using the principle shown in embodiment 300, and then the network parameters of the network layer are converted between floating point numbers and fixed point numbers according to the scaling factor, thereby achieving quantization or pre-quantization of the network parameters.

[0081] According to an embodiment of the present disclosure, when quantizing a distributed deployment model, in addition to quantizing the network parameters of the network layer, for example, a scaling factor of the activation value can also be determined, and the scaling factor is derived together with the network parameters of the quantized network layer to obtain a quantized model that can be put into use online. The activation value can be an input activation value or an output activation value of the network layer.

[0082] In one embodiment, it is necessary to determine the activation value of the scaling factor as the input activation value to unify the format of the network parameters and input data of each network layer, so that each network layer can operate the input data according to the network parameters. For example, the principle of determining the scaling factor of the input activation value can be similar to the principle of determining the scaling factor of the network parameters in the network layer described above.

[0083] For example, the computing node can use the deployed model to process the aforementioned predetermined calibration data to obtain the input activation value of each network layer. Subsequently, the scaling factor of the input activation value of each network layer is determined according to the target quantization accuracy and the input activation value of each network layer. Among them, the target quantization accuracy is the quantization accuracy based on which the scaling factor of the network parameters for each network layer is determined. For example, if the first quantization accuracy is used to quantize each network layer, then when determining the scaling factor of the input activation value of each network layer, the target quantization accuracy is the first quantization accuracy.

[0084] For example, the deployed model can also be used to process N predetermined calibration data separately to obtain N groups of input activation values ​​for each network layer. Subsequently, a scaling factor is determined according to each group of input activation values ​​and the target quantization accuracy, and a total of N scaling factors are obtained. Finally, this embodiment can comprehensively consider the obtained N scaling factors to determine the third scaling factor of the input activation value of each network layer. For example, this embodiment can use the average value of the N scaling factors as the third scaling factor. Wherein, N is a natural number greater than 1. By determining the third scaling factor based on N predetermined calibration data, the third scaling factor that exists in the third scaling factor determined only based on a single predetermined calibration data can be compensated for, and the defect of affecting the quantization accuracy can be improved, and thus the accuracy of the determined third scaling factor can be improved. This is because the value of the input activation value of each network layer will be different for different data. However, it should be noted that the value of N should not be too large, because too much predetermined calibration data will increase the probability of abnormal outliers appearing in the input activation value of each network layer, which will affect the quantization accuracy of the model. For example, N can be set to 10 or other values ​​selected according to actual needs, and the present disclosure does not limit this.

[0085] Figure 5 3 is a schematic diagram of the principle of determining the scaling factor of the input activation value of the self-attention layer according to an embodiment of the present disclosure.

[0086] According to an embodiment of the present disclosure, for a self-attention layer that needs to perform a matrix multiplication operation, the scaling factor of the input activation value of the self-attention layer determined by multiple computing nodes can also be limited to a unified scaling factor to improve the accuracy of the quantized model. This is because in the process of realizing the concept of the present disclosure, the inventors found that for different computing nodes, due to the use of different training samples, the distribution law of the input activation value of the self-attention layer is inconsistent, and the input activation value of the self-attention layer is usually relatively discrete, the degree of outliers is large and the proportion of outliers is high, which is not suitable for quantization. By making multiple computing nodes adopt a unified scaling factor, more distribution situations of the input activation value of the self-attention layer can be taken into account, which is conducive to improving the accuracy of the adopted scaling factor, thereby reducing the loss of the quantization model.

[0087] In one embodiment, multiple computing nodes may send the determined third scaling factors for the input activation values ​​of the self-attention layer to the target node, and the target node summarizes the received multiple third scaling factors, thereby determining a unified scaling factor for the input activation values ​​of the self-attention layer according to the multiple third scaling factors, and sending the unified scaling factor to the multiple computing nodes. Then, the multiple computing nodes may use the unified scaling factor to replace the originally determined third scaling factor for the input activation values ​​of the self-attention layer.

[0088] For example, the target node may use the average value of multiple third scaling factors as the unified scaling factor. Alternatively, the multiple third scaling factors may be clustered and the determined cluster center may be used as the unified scaling factor.

[0089] like Figure 5 As shown, in one embodiment 500, each computing node 510 of the plurality of computing nodes may send a third scaling factor for the input activation value of the self-attention layer to the target node through operation S511.

[0090] After receiving the third scaling factors sent by multiple computing nodes, the target node may, for example, first perform operation S521 to determine the value range of the received multiple third scaling factors. For example, the received multiple third scaling factors include scaling factors a, b, c, d arranged from large to small, and the value range can be determined to be [d, a]. Subsequently, the target node may perform operation S522 to take multiple values ​​from the value range according to a predetermined value interval as multiple candidate scaling factors. For example, the predetermined value interval may be 0.01, and may start from the minimum value d and take a value at each predetermined value interval. The multiple candidate scaling factors finally obtained include the lower limit value of the value range, and may also include the upper limit value of the value range according to actual needs. Subsequently, the target node may send the multiple candidate scaling factors to multiple computing nodes through operation S523.

[0091] After receiving the multiple candidate scaling factors, each computing node may, for example, perform operation S512 to determine a loss value for quantizing the input activation value of the self-attention layer according to each of the candidate scaling factors as a quantized loss value for each of the candidate scaling factors. For example, a principle similar to the principle of determining the quantized loss value of each network layer described above may be used to determine the quantized loss value for each of the candidate scaling factors.

[0092] For example, each computing node can use the deployed model to process the predetermined calibration data to obtain the input activation value of the self-attention layer. At the same time, the network parameters of the self-attention layer can be pre-quantized using the aforementioned target quantization accuracy to obtain the pre-quantized self-attention layer. After obtaining the input activation value, each alternative scaling factor can be used to pre-quantize the input activation value of the self-attention layer to obtain the pre-quantized activation value. After obtaining the input activation value, the self-attention layer before quantization can be used to process the input activation value to obtain a third output activation value. After obtaining the pre-quantized self-attention layer and the pre-quantized activation value, the pre-quantized self-attention layer can be used to process the pre-quantized activation value to obtain a fourth output activation value. Finally, each computing node can determine the loss value of the input activation value of the self-attention layer quantized according to each alternative scaling factor based on the difference between the third output activation value and the fourth output activation value. It should be noted that the above-mentioned principle of pre-quantization of network parameters is similar to that described above, and the principle of quantizing the input activation value is also similar to that described above, which will not be repeated here.

[0093] For example, the square value of the difference between the corresponding elements in the third output activation value and the fourth output activation value can be calculated first, and the sum of multiple square values ​​can be used as the quantization loss value for each candidate scaling factor. Alternatively, any distance between two output activation values ​​(such as Euclidean distance, bulldozer distance, etc.) can be used as the quantization loss value for each candidate scaling factor.

[0094] After obtaining the quantization loss value for each candidate scaling factor, each computing node 510 may perform operation S513 to send the quantization loss value for each candidate scaling factor to the target node 520 .

[0095] After receiving the quantization loss values ​​for each candidate scaling factor sent by the multiple computing nodes, the target node 520 may, for example, perform operation S524 according to the received multiple quantization loss values ​​for each candidate scaling factor to determine the target quantization loss value for each candidate scaling factor. For example, the sum of the multiple quantization loss values ​​may be used as the target quantization loss value. Alternatively, the average value of the multiple quantization loss values ​​may be used as the target quantization loss value, which is not limited in the present disclosure.

[0096] After obtaining multiple target quantization loss values ​​for multiple candidate scaling factors, the target node 520 may, for example, perform operation S525 to use the candidate scaling factor with the smallest target quantization loss value among the multiple candidate scaling factors as the target scaling factor, and perform operation S526 to send the target scaling factor to the multiple computing nodes. After receiving the target scaling factor, the multiple computing nodes may perform operation S514 to use the target scaling factor as the scaling factor for the input activation value of the self-attention layer.

[0097] It is understandable that after obtaining the scaling factor of the input activation value for each network layer and quantizing the network parameters of the network layer, the neural network model can be exported according to actual needs. For example, if the neural network model is exported in the format of an onnx file, for example, for input activation values ​​without weights, an analog quantization layer and an inverse quantization layer can be inserted before the network layer that needs to be quantized, and the analog quantization layer and the inverse quantization layer both store the scaling factor for the input activation value. For input activation values ​​with weights, an inverse quantization layer can be inserted before the network layer that needs to be quantized, and the inverse quantization layer stores the scaling factor for the input activation value. Among them, the weighted input activation value refers to the need to load the weight from the local, and use the weight and the input activation value together as the input activation value. When the neural network model is put into use online, the aforementioned added inverse quantization layer can be removed.

[0098] Figure 6 It is a schematic diagram of the application of the quantization method of the distributed deployment model according to an embodiment of the present disclosure.

[0099] like Figure 6 As shown, when the quantization method of the distributed deployment model provided by the present disclosure is applied to a neural network model based on the Transformer architecture, the embodiment 600 can, for example, determine the embedding layer Q 602 for generating the query feature Query, the embedding layer K 603 for generating the key feature Key, and the embedding layer V 604 for generating the value feature Value in the Transformer architecture, the Projection layer 608 for performing matrix multiplication on the self-attention operation result, the first feedforward neural network layer FFN1 610, and the second feedforward neural network layer FFN2 612 as target network layers. These target network layers are layers that need to calculate matrix multiplication, and by quantizing according to the first quantization accuracy, the scale of the model can be effectively reduced. The remaining layers in the Transformer architecture (e.g., normalization layer LayerNorm1 601, layer BMM1 605 for calculating the attention matrix, softmax activation layer 606, layer BMM2607 for weighting value features, normalization layer LayerNorm2609, activation layer GELU 611) can be quantized according to the second quantization precision, or the network weights of these layers can be kept in floating point format, which is not limited in the present disclosure.

[0100] The model quantized by this embodiment can be deployed on a single A100 chip, for example. Compared with the model before quantization, the calculation rate can be accelerated by at least 1.2 times.

[0101] It should be noted that, in order to facilitate the calculation, it is necessary to insert a quantization layer or an inverse quantization node between two layers with different quantization precisions, so as to quantize or inversely quantize the activation value of each network layer according to actual needs, which will not be described in detail here. The original format of the network parameters in the neural network model can be, for example, a floating point format such as float32 format or float16 format, which is not limited in this disclosure.

[0102] By adopting the quantization method of the distributed deployment model provided by the present invention, since the target network layer can be determined according to the quantization loss value, and the scaling factor of the input activation value of the input self-attention layer can be flexibly determined, automatic optimization of model quantization without human intervention can be achieved, which can largely ensure that the accuracy of the quantized model is basically unchanged compared to the pre-quantized model.

[0103] Based on the quantization method of the distributed deployment model provided by the present disclosure, the present disclosure also provides a quantization device of the distributed deployment model. Figure 7 The device is described in detail.

[0104] Figure 7 It is a structural block diagram of a quantization device of a distributed deployment model according to an embodiment of the present disclosure.

[0105] like Figure 7 As shown, the quantization device 700 of the distributed deployment model of this embodiment may include a first loss determination module 710 , a first loss sending module 720 and a parameter quantization module 730 .

[0106] The first loss determination module 710 is used to determine, for the multiple network layers included in the deployed model, a loss value of each network layer in the multiple network layers quantized according to the first quantization accuracy as the quantization loss value of each network layer. In one embodiment, the first loss determination module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0107] The first loss sending module 720 is used to send the quantized loss values ​​of the multiple network layers to the target node among the multiple computing nodes of the distributed deployment model, so that the target node determines the target network layer among the multiple network layers according to the received quantized loss value. In one embodiment, the first loss sending module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0108] The parameter quantization module 730 is used to quantize the network parameters of the target network layer according to the first quantization accuracy. In one embodiment, the parameter quantization module 730 can be used to perform the operation S230 described above, which will not be described in detail here.

[0109] According to an embodiment of the present disclosure, the target node is any node specified among multiple computing nodes, and the above-mentioned device 700 may also include: a target loss determination module, which is used to determine the maximum value of multiple quantization loss values ​​of each network layer in response to receiving the quantization loss values ​​of multiple network layers sent by multiple computing nodes, as the target quantization loss value of each network layer; a target layer determination module, which is used to determine the network layer whose target quantization loss value among multiple network layers is less than the predetermined loss value, and obtain the target network layer.

[0110] According to an embodiment of the present disclosure, the above-mentioned first loss determination module 710 may include an absolute value determination submodule, which is used to determine the absolute value of the values ​​of the network parameters included in each network layer; a scaling factor determination submodule, which is used to determine the first scaling factor of the network parameters for each network layer based on the maximum value of the first quantization accuracy and the absolute value; and a first loss determination submodule, which is used to determine the loss value of the network parameters of each network layer quantized according to the first scaling factor, as the quantization loss value of each network layer.

[0111] According to an embodiment of the present disclosure, a first loss determination submodule may include: a prequantization unit, used to prequantize the network parameters of each network layer according to a first scaling factor to obtain a prequantized network layer; an activation value acquisition unit, used to process predetermined calibration data using a deployed model to obtain an input activation value for each network layer; a first activation processing unit, used to process the input activation value using each network layer to obtain a first output activation value; a second activation processing unit, used to process the input activation value using the prequantized network layer to obtain a second output activation value; and a loss value determination unit, used to determine the quantization loss value of each network layer based on the distance between the first output activation value and the second output activation value.

[0112] According to an embodiment of the present disclosure, the above-mentioned device 700 may also include a first factor determination module, which is used to determine, for other network layers in the multiple network layers except the target network layer, a second scaling factor for the network parameters of other network layers according to the second quantization precision and the values ​​of the network parameters included in the other network layers. The above-mentioned parameter quantization module 730 may also be used to quantize the network parameters of other network layers according to the second scaling factor, wherein the first quantization precision is lower than the second quantization precision.

[0113] According to an embodiment of the present disclosure, the above-mentioned device 700 may also include: an activation value determination module, which is used to use the deployed model to process N predetermined calibration data respectively to obtain N groups of input activation values ​​for each network layer; a second factor determination module, which is used to determine the scaling factor for each group of input activation values ​​according to the target quantization accuracy and each group of input activation values ​​in the N groups of input activation values; a third factor determination module, which is used to determine the third scaling factor for the input activation value of each network layer according to the N scaling factors for the N groups of input activation values, wherein the target quantization accuracy is the quantization accuracy based on which the scaling factor of the network parameters for each network layer is determined; and N is a natural number greater than 1.

[0114] According to an embodiment of the present disclosure, multiple network layers include a self-attention layer. The above-mentioned device 700 may also include: a first factor sending module, which is used to send a third scaling factor for the input activation value of the self-attention layer to the target node, so that the target node determines a plurality of alternative scaling factors for the input activation value of the self-attention layer according to the received third scaling factor; a second loss determination module, which is used to determine, in response to receiving a plurality of alternative scaling factors, a loss value for quantizing the input activation value of the self-attention layer according to each alternative scaling factor as a quantized loss value for each alternative scaling factor; a second loss sending module, which is used to send the quantized loss value for each alternative scaling factor to the target node, so that the target node determines a target scaling factor among the plurality of alternative scaling factors; and a fourth factor determination module, which is used to use the target scaling factor as a scaling factor for the input activation value of the self-attention layer.

[0115] According to an embodiment of the present disclosure, the target node is any node specified in a plurality of computing nodes. The above-mentioned device 700 may also include: a value interval determination module, which is used to determine the value interval of a plurality of third scaling factors in response to receiving a plurality of third scaling factors sent by a plurality of computing nodes; a value module, which is used to take a plurality of values ​​from the value interval according to a predetermined value interval as a plurality of candidate scaling factors; a second factor sending module, which is used to send a plurality of candidate scaling factors to a plurality of computing nodes; a third loss determination module, which is used to determine the target quantization loss value for each candidate scaling factor according to a plurality of quantization loss values ​​for each candidate scaling factor in response to receiving a quantization loss value for each candidate scaling factor sent by a plurality of computing nodes; and a target factor determination module, which is used to determine the candidate scaling factor with the smallest target quantization loss value among the plurality of candidate scaling factors as the target scaling factor.

[0116] According to an embodiment of the present disclosure, the above-mentioned second loss determination module may include: an activation value acquisition submodule, which is used to process the predetermined calibration data using the deployed model to obtain the input activation value of the self-attention layer; a first prequantization submodule, which is used to prequantize the network parameters of the self-attention layer according to the target quantization accuracy to obtain the prequantized self-attention layer; a second prequantization submodule, which is used to prequantize the input activation value of the self-attention layer using each alternative scaling factor to obtain the prequantized activation value; a first activation processing submodule, which is used to process the input activation value of the self-attention layer using the self-attention layer to obtain a third output activation value; a second activation processing submodule, which is used to process the prequantized activation value using the prequantized self-attention layer to obtain a fourth output activation value; and a second loss determination submodule, which is used to determine the loss value of the input activation value of the self-attention layer quantized according to each alternative scaling factor based on the third output activation value and the fourth output activation value.

[0117] According to an embodiment of the present disclosure, the above-mentioned loss value determination unit may include: a distance determination subunit, used to determine the Euclidean distance between the first output activation value and the second output activation value and the bulldozing distance between the first output activation value and the second output activation value; and a loss determination subunit, used to determine the quantization loss value of each network layer according to the Euclidean distance and the bulldozing distance.

[0118] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0120] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement the quantization method of the distributed deployment model of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0121] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0122] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the quantization method of the distributed deployment model. For example, in some embodiments, the quantization method of the distributed deployment model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the quantization method of the distributed deployment model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the quantization method of the distributed deployment model in any other appropriate manner (e.g., by means of firmware).

[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A quantification method for a distributed deployment model, comprising: For a plurality of network layers included in the deployed model, determining a loss value of each network layer in the plurality of network layers quantized according to a first quantization precision as a quantization loss value of each network layer; the model includes a text processing model, an image processing model, or a speech processing model; Sending the quantized loss values ​​of the multiple network layers to a target node among the multiple computing nodes of the distributed deployment model, so that the target node determines a target network layer among the multiple network layers according to the received quantized loss values; as well as quantizing the network parameters of the target network layer according to the first quantization accuracy; The method further includes: determining a scaling factor of the input activation value of each network layer according to a target quantization accuracy and an input activation value of each network layer, wherein the target quantization accuracy is the quantization accuracy based on which the network parameters of each network layer are quantized; Wherein, the multiple network layers include a self-attention layer; and the method further includes: Sending the determined scaling factor of the input activation value of the self-attention layer to the target node; determining, by the target node, a target scaling factor for the input activation value of the self-attention layer based on the received scaling factor; and The target scaling factor is used as a scaling factor for the input activation values ​​of the self-attention layer.

2. The method according to claim 1, wherein: The target node is any node specified in the multiple computing nodes; the method further includes: In response to receiving the quantization loss values ​​of the multiple network layers sent by the multiple computing nodes, determining a maximum value among the multiple quantization loss values ​​of each network layer as a target quantization loss value of each network layer; and Determine a network layer among the multiple network layers whose target quantization loss value is less than a predetermined loss value to obtain the target network layer.

3. The method according to claim 1, wherein: Determining a loss value of each network layer in the plurality of network layers quantized according to a first quantization accuracy as the quantization loss value of each network layer includes: Determine the absolute value of the network parameter included in each network layer; Determining a first scaling factor for the network parameter of each network layer according to the maximum value of the first quantization precision and the absolute value; and A loss value of quantizing a network parameter of each network layer according to the first scaling factor is determined as a quantization loss value of each network layer.

4. The method according to claim 3, wherein: The determining a loss value of a network parameter of each network layer quantized according to the first scaling factor as the quantized loss value of each network layer comprises: Pre-quantizing the network parameters of each network layer according to the first scaling factor to obtain a pre-quantized network layer; Processing predetermined calibration data using the deployed model to obtain input activation values ​​for each network layer; Using each of the network layers to process the input activation value to obtain a first output activation value; Processing the input activation value using the pre-quantized network layer to obtain a second output activation value; and Determine a quantization loss value of each network layer according to a distance between the first output activation value and the second output activation value.

5. The method according to claim 1, further comprising: For other network layers among the multiple network layers except the target network layer, determining a second scaling factor for the network parameters of the other network layers according to a second quantization precision and values ​​of the network parameters included in the other network layers; as well as quantizing the network parameters of the other network layers according to the second scaling factor, The first quantization accuracy is lower than the second quantization accuracy.

6. The method according to claim 1, wherein: Determining the scaling factor of the input activation value of each network layer according to the target quantization accuracy and the input activation value of each network layer includes: Using the deployed model to process N predetermined calibration data respectively, to obtain N groups of input activation values ​​for each network layer; Determining a scaling factor for each set of input activation values ​​according to a target quantization accuracy and each set of input activation values ​​in the N sets of input activation values; and Determining a third scaling factor for the input activation values ​​of each network layer according to the N scaling factors for the N groups of input activation values, Wherein, N is a natural number greater than 1.

7. The method according to claim 1, further comprising: In response to receiving the plurality of candidate scaling factors, determining a loss value for quantizing an input activation value of the self-attention layer according to each candidate scaling factor as a quantized loss value for each candidate scaling factor; The plurality of candidate scaling factors are determined by the target node based on the received scaling factors; as well as The quantization loss value for each candidate scaling factor is sent to the target node, so that the target node determines a target scaling factor from the plurality of candidate scaling factors.

8. The method according to claim 7, wherein: The target node is any node specified in the multiple computing nodes; the method further includes: In response to receiving a plurality of scaling factors sent by the plurality of computing nodes, determining value intervals of the plurality of scaling factors; Taking a plurality of values ​​from the value interval according to a predetermined value interval as a plurality of candidate scaling factors; sending the plurality of candidate scaling factors to the plurality of computing nodes; In response to receiving the quantization loss values ​​for each candidate scaling factor sent by the multiple computing nodes, determining a target quantization loss value for each candidate scaling factor according to the multiple quantization loss values ​​for each candidate scaling factor; and The candidate scaling factor with the smallest target quantization loss value among the multiple candidate scaling factors is determined as the target scaling factor.

9. The method according to claim 7, wherein: Determining a loss value for quantizing the input activation value of the self-attention layer according to each candidate scaling factor includes: Processing predetermined calibration data using the deployed model to obtain input activation values ​​of the self-attention layer; Pre-quantizing the network parameters of the self-attention layer according to the target quantization accuracy to obtain a pre-quantized self-attention layer; Pre-quantizing the input activation value of the self-attention layer using each of the candidate scaling factors to obtain a pre-quantized activation value; Processing the input activation value of the self-attention layer using the self-attention layer to obtain a third output activation value; Processing the prequantized activation value using the prequantized self-attention layer to obtain a fourth output activation value; and A loss value for quantizing an input activation value of the self-attention layer according to each candidate scaling factor is determined based on the third output activation value and the fourth output activation value.

10. The method according to claim 4, wherein: The determining, according to the distance between the first output activation value and the second output activation value, the quantization loss value of each network layer comprises: determining a Euclidean distance between the first output activation value and the second output activation value and a bulldozing distance between the first output activation value and the second output activation value; and According to the Euclidean distance and the bulldozer distance, a quantization loss value of each network layer is determined.

11. A quantification device for a distributed deployment model, comprising: A first loss determination module is used to determine, for a plurality of network layers included in a deployed model, a loss value of each network layer in the plurality of network layers quantized according to a first quantization accuracy as a quantization loss value of each network layer, wherein the model includes a text processing model, an image processing model, or a speech processing model; A first loss sending module, configured to send the quantized loss values ​​of the multiple network layers to a target node among the multiple computing nodes of the distributed deployment model, so that the target node determines a target network layer among the multiple network layers according to the received quantized loss value; as well as A parameter quantization module, configured to quantize the network parameters of the target network layer according to the first quantization accuracy; The device further includes a first factor determination module, which is used to determine a scaling factor of the input activation value of each network layer according to a target quantization accuracy and an input activation value of each network layer, wherein the target quantization accuracy is the quantization accuracy based on which the network parameters of each network layer are quantized; Wherein, the plurality of network layers include a self-attention layer; and the device further includes: A first factor sending module is used to send the determined scaling factor of the input activation value of the self-attention layer to the target node; so that the target node determines the target scaling factor for the input activation value of the self-attention layer based on the received scaling factor; and The second factor determination module is used to use the target scaling factor as a scaling factor for the input activation value of the self-attention layer.

12. The device according to claim 11, wherein The target node is any node specified in the multiple computing nodes; the device also includes: a target loss determination module, configured to determine, in response to receiving the quantization loss values ​​of the multiple network layers sent by the multiple computing nodes, a maximum value among the multiple quantization loss values ​​of each network layer as a target quantization loss value of each network layer; The target layer determination module is used to determine the network layer whose target quantization loss value is less than the predetermined loss value among the multiple network layers, and obtain the target network layer.

13. The device according to claim 11, wherein: The first loss determination module includes: An absolute value determination submodule, used to determine the absolute value of the value of the network parameter included in each network layer; a first factor determination submodule, configured to determine a first scaling factor of a network parameter for each network layer according to a maximum value of the first quantization precision and the absolute value; and The first loss determination submodule is used to determine a loss value of a network parameter of each network layer quantized according to the first scaling factor as a quantization loss value of each network layer.

14. The device according to claim 13, wherein: The first loss determination submodule includes: A prequantization unit, configured to prequantize the network parameters of each network layer according to the first scaling factor to obtain a prequantized network layer; An activation value obtaining unit, used to process predetermined calibration data using the deployed model to obtain an input activation value input to each network layer; A first activation processing unit, configured to process the input activation value using each network layer to obtain a first output activation value; A second activation processing unit, configured to process the input activation value using the pre-quantized network layer to obtain a second output activation value; and A loss value determining unit is used to determine the quantization loss value of each network layer according to the distance between the first output activation value and the second output activation value.

15. The apparatus according to claim 11, further comprising: A third factor determination module, configured to determine, for other network layers among the multiple network layers except the target network layer, a second scaling factor for the network parameters of the other network layers according to the second quantization precision and the values ​​of the network parameters included in the other network layers; as well as The parameter quantization module is further used to: quantize the network parameters of the other network layers according to the second scaling factor, The first quantization accuracy is lower than the second quantization accuracy.

16. The device according to claim 11, wherein The first factor determination module includes: An activation value determination submodule, used to process N predetermined calibration data respectively using the deployed model to obtain N groups of input activation values ​​for each network layer; A second factor determination submodule is used to determine a scaling factor for each group of input activation values ​​according to a target quantization accuracy and each group of input activation values ​​in the N groups of input activation values; A third factor determination submodule is configured to determine a third scaling factor for the input activation value of each network layer according to the N scaling factors for the N groups of input activation values, Wherein, N is a natural number greater than 1.

17. The device according to claim 11, wherein: The device also includes: A second loss determination module is configured to determine, in response to receiving the plurality of candidate scaling factors, a loss value for quantizing the input activation value of the self-attention layer according to each candidate scaling factor as a quantized loss value for each candidate scaling factor; the plurality of candidate scaling factors are determined by the target node according to the received scaling factors; The second loss sending module is used to send the quantization loss value for each candidate scaling factor to the target node, so that the target node determines the target scaling factor among the multiple candidate scaling factors.

18. The device according to claim 17, wherein: The target node is any node specified in the multiple computing nodes; the device also includes: a value interval determination module, configured to determine the value intervals of the plurality of scaling factors in response to receiving the plurality of third scaling factors sent by the plurality of computing nodes; A value taking module, used for taking a plurality of values ​​from the value taking interval according to a predetermined value taking interval as a plurality of candidate scaling factors; A second factor sending module, configured to send the plurality of candidate scaling factors to the plurality of computing nodes; a third loss determination module, configured to determine, in response to receiving the quantization loss values ​​for each candidate scaling factor sent by the multiple computing nodes, a target quantization loss value for each candidate scaling factor according to the multiple quantization loss values ​​for each candidate scaling factor; and The target factor determination module is used to determine the candidate scaling factor with the smallest target quantization loss value among the multiple candidate scaling factors as the target scaling factor.

19. The device according to claim 17, wherein: The second loss determination module includes: An activation value obtaining submodule, used to process predetermined calibration data using the deployed model to obtain an input activation value of the self-attention layer; A first prequantization submodule, configured to prequantize the network parameters of the self-attention layer according to the target quantization accuracy to obtain a prequantized self-attention layer; A second prequantization submodule is used to prequantize the input activation value of the self-attention layer using each candidate scaling factor to obtain a prequantized activation value; A first activation processing submodule, configured to process an input activation value of the self-attention layer using the self-attention layer to obtain a third output activation value; A second activation processing submodule, configured to process the prequantized activation value using the prequantized self-attention layer to obtain a fourth output activation value; and The second loss determination submodule is used to determine, based on the third output activation value and the fourth output activation value, a loss value for quantizing the input activation value of the self-attention layer according to each alternative scaling factor.

20. The device according to claim 14, wherein: The loss value determination unit comprises: a distance determination subunit, configured to determine a Euclidean distance between the first output activation value and the second output activation value and a bulldozing distance between the first output activation value and the second output activation value; and The loss determination subunit is used to determine the quantization loss value of each network layer according to the Euclidean distance and the bulldozer distance.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is stored on at least one of a readable storage medium and an electronic device, and when the computer program / instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images