Quantization policy determination method, model quantization system and related device

Abstract

Provided in the present application are a quantization policy determination method, a model quantization system and a related device. The method comprises the following steps: performing free combination on model-supported objects to be quantized and quantizable data types to obtain a plurality of different quantization policies; then determining the sensitivities of the different quantization policies on the basis of a sensitivity algorithm, the sensitivities being used for indicating effects of the quantization policies on the precision of a model to be quantized; determining a first quantization policy on the basis of the sensitivities of the quantization policies; if the first quantization policy passes quantization evaluation, using the first quantization policy to quantize said model, such that the entire quantization process does not require quantization search; and if the quantization evaluation result is fail, improving, by means of quantization fallback, the precision of some objects to be quantized in the first quantization policy until a second quantization policy that passes the evaluation is obtained. Compared with traditional methods of performing quantization search on the basis of large-scale parameters, the method can reduce the search space and improve the efficiency of model quantization.

Description

Method for determining quantization strategy, model quantization system and related device

[0001] The present application claims priority to the Chinese patent application No. 202411376693.4, filed on September 29, 2024, with the State Intellectual Property Office of China, and entitled "Method for determining quantization strategy, model quantization system and related device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI), and in particular to a method for determining quantization strategy, a model quantization system and related device. BACKGROUND

[0003] With the continuous development of technology, the size of large language models (LLMs) is getting larger and larger, and the inference latency of large language models is also getting longer and longer. In order to reduce the storage overhead and inference latency of the model, model quantization technology emerges as the times require. Model quantization refers to the process of converting high-precision parameters in a large language model into low-precision parameters, or converting high-precision calculations into low-precision calculations. By reducing the precision of model parameters or calculations, the storage requirements and computational complexity of a large language model can be reduced, thereby reducing the storage overhead and inference latency of the large language model.

[0004] However, model quantization technology changes the precision of model parameters and calculations, which may cause precision loss for large language models. The precision loss of different large language models varies, and therefore each large language model needs to determine a targeted quantization strategy to ensure that the performance loss after model quantization is within an acceptable range. Generally, a search algorithm is used to determine the optimal quantization strategy for a large language model, and then a corresponding quantization operation is performed. However, due to the large number of parameters and large amount of calculations of a large language model, the search space of the search algorithm is very large, and the search time increases exponentially with the size of the search space, resulting in low efficiency of large language model quantization. SUMMARY

[0005] The present application provides a method for determining quantization strategy, a model quantization system and related device, which are used to solve the problem of large search space and long search time of model quantization strategy, resulting in low efficiency of model quantization.

[0006] In a first aspect, a method for determining a quantization strategy is provided. The method is applied to a model quantization system and includes the following steps: obtaining one or more objects to be quantized of a model to be quantized and at least one data type that each object to be quantized can be quantized to; determining a plurality of quantization strategies based on the at least one data type that each object to be quantized can be quantized to, wherein each quantization strategy includes a quantization data type of each object to be quantized, and the quantization data type is one of the at least one data type that each object to be quantized can be quantized to; determining a sensitivity of each quantization strategy, wherein the sensitivity is used to indicate a degree of accuracy loss of the model to be quantized after using the quantization strategy for quantization; and obtaining a first quantization strategy from the plurality of quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize the one or more objects to be quantized.

[0007] By implementing the method described in the first aspect, the objects to be quantized of the model to be quantized and the data types that can be quantized are freely combined to obtain a plurality of different quantization strategies, each of which has different data types corresponding to the objects to be quantized. Then, the sensitivity of each quantization strategy is determined based on a sensitivity algorithm, which is used to indicate the influence of the quantization strategy on the accuracy of the model to be quantized. According to the sensitivity of each quantization strategy, the first quantization strategy is determined, so that the entire quantization process does not need to be searched for quantization, thereby solving the problem of low quantization efficiency caused by long time consumption of the model quantization due to quantization search.

[0008] In a possible implementation, when determining the sensitivity of each quantization strategy, the sensitivity of each object to be quantized in each quantization strategy can be determined, and the sensitivity of each quantization strategy is obtained based on the sensitivity of each object to be quantized, wherein the sensitivity of each object to be quantized is obtained based on one or more parameters of the average trace of the hessian matrix of the object to be quantized, and the loss value before and after quantization of the object to be quantized.

[0009] It should be understood that the smaller the average trace of the hessian matrix of the weight, the greater the accuracy loss caused by the quantization of the weight to the entire model. Therefore, by taking the average trace as the sensitivity parameter, the influence of the quantization strategy on the accuracy of the model can be well indicated. Each object to be quantized can be understood as a weight parameter or an activation value during quantization, and the corresponding sensitivity can be calculated according to this principle. The sensitivity of each object to be quantized can be used to obtain the sensitivity of the entire quantization strategy.

[0010] The above implementation manner determines the sensitivity of the entire quantization strategy by calculating the sensitivity of each to-be-quantized object, and then screens out the best first quantization scheme according to the sensitivity of the entire quantization strategy. In this way, the precision is evaluated in units of quantization strategies, which can avoid the problem that the model precision exceeds the controllable range due to the accumulation of precision caused by quantization of multiple to-be-quantized objects, so that the entire model can quantize the to-be-quantized objects in multiple calculation stages, and the quantization degree of the model can be maximized, and the potential of overall model quantization is developed.

[0011] In a possible implementation manner, after obtaining the first quantization scheme, the first quantization strategy can be subjected to quantization evaluation to obtain a quantization evaluation result. In a case where the quantization evaluation result is unqualified, the first quantization strategy is subjected to quantization rollback, the quantization data type of the to-be-quantized object in the first quantization strategy is modified, a second quantization strategy is obtained, and the second quantization strategy is used to quantize one or more to-be-quantized objects.

[0012] The above implementation manner subjects the first quantization strategy to quantization evaluation. If the evaluation is qualified, the first quantization strategy is determined as the quantization strategy of the to-be-quantized model, which can obtain a qualified quantization strategy without quantization search, and ensures the efficiency and accuracy of quantization strategy determination. If the evaluation is unqualified, the first quantization strategy is subjected to quantization rollback, the precision of part of the to-be-quantized objects in the first quantization strategy is improved, and a second quantization strategy that is qualified in evaluation is obtained. Compared with the traditional quantization search based on a large number of parameters, the manner of determining the first quantization strategy and then performing rollback can reduce the search space, thereby solving the problem of low model quantization efficiency caused by the large search space and long search time when searching for a quantization strategy in the model quantization process.

[0013] In a possible implementation manner, the method further includes the following steps: quantizing the to-be-quantized model using the first quantization strategy or the second quantization strategy to obtain a quantized model, and sending the quantized model to an inference system.

[0014] It should be noted that, in a case where the quantization evaluation result is qualified, the first quantization strategy can be used for model quantization; in a case where the quantization evaluation result is unqualified, the first quantization strategy is subjected to quantization rollback to obtain a second quantization strategy according to the manner described above, and then the second quantization strategy is used for model quantization.

[0015] Further, after determining the second quantization strategy, the model quantization system can feed back the scheme to the client, and the user can choose whether to use the scheme to quantize the model, or the model quantization system can directly use the second quantization strategy to quantize the model to obtain quantized model data, and then export the quantization result to the client, and the user can choose whether to deploy the model data in the inference system, or the model quantization system can use the second quantization strategy to quantize the model to obtain quantized model data, and then automatically deploy the model data in the inference system, simplifying the user operation process. According to actual business needs, the specific implementation is not limited in the present application.

[0016] In a possible implementation, the model to be quantized includes a plurality of network layers, and in the first quantization strategy, the quantization data types of the same to-be-quantized object in each network layer are the same; in the second quantization strategy, the quantization data types of the same to-be-quantized object in each network layer are the same or different, and the number of to-be-quantized objects in the second quantization strategy is less than the number of to-be-quantized objects in the first quantization strategy.

[0017] The above implementation first determines the sensitivity of different quantization strategies, and coarsely selects a relatively excellent first quantization strategy, in which the quantization data types of the same to-be-quantized object in each network layer are the same. When the first quantization strategy is determined to be unqualified through quantization evaluation, quantization rollback is performed according to each network layer, and a more fine-grained second quantization strategy can be obtained. Compared with the traditional way of searching for corresponding quantization data types for a large number of parameters in sequence, the above way of coarse selection and fine rollback can reduce the search space, thereby improving the efficiency of large language model quantization.

[0018] In a possible implementation, when the first quantization strategy is quantization-rolled back to obtain the second quantization strategy, a plurality of rollback schemes can be first obtained, each rollback scheme including at least one to-be-quantized object, different rollback schemes including different numbers of to-be-quantized objects, the quantization data types of each to-be-quantized object in a first rollback scheme are determined, a third quantization strategy corresponding to the first rollback scheme is obtained, the third quantization strategy is quantization-evaluated to obtain a rollback evaluation result, in the case that the rollback evaluation result is qualified, the third quantization strategy is determined as the second quantization strategy, in the case that the rollback evaluation result is unqualified, the quantization data types of each to-be-quantized object in a second rollback scheme are determined, a fourth quantization strategy corresponding to the second rollback scheme is obtained, and the number of to-be-quantized objects in the first rollback scheme is not less than the number of to-be-quantized objects in the second rollback scheme.

[0019] For example, the fallback solution 1 includes the to-be-quantized object 1 and the to-be-quantized object 2, the fallback solution 2 includes the to-be-quantized object 1, the to-be-quantized object 2 and the to-be-quantized object 3, the fallback solution 3 includes the to-be-quantized object 1 and the to-be-quantized object 3, and so on. Then, the optimal quantization strategy under each fallback solution is determined in sequence according to the search order, and the optimal quantization strategy is quantization evaluated. If the evaluation result is qualified, the quantization strategy is determined as the second quantization strategy. If not, the optimal quantization strategy under the next fallback solution is determined, and so on, until the second quantization strategy with a qualified evaluation result is obtained.

[0020] The above implementation manner pre-establishes fallback solutions including different to-be-quantized objects, then determines the optimal third quantization strategy under the first fallback solution according to the search order of gradually reducing the number of to-be-quantized objects, and then evaluates the optimal third quantization strategy. If the evaluation result is not qualified, the optimal quantization strategy under the next fallback solution is determined according to the search order, until the qualified quantization strategy is obtained. If there are M fallback solutions, the second quantization strategy with a qualified evaluation result can be obtained by evaluating at most M times, thereby further improving the efficiency of quantization fallback.

[0021] In a possible implementation manner, when the third quantization strategy corresponding to the first fallback solution is obtained, the quantization data type of each to-be-quantized object in the first fallback solution in the plurality of fallback solutions can be determined, the at least one hierarchical quantization strategy of each network layer under the first fallback solution can be obtained based on the to-be-quantized objects included in the first fallback solution and the at least one data type that can be quantized for each to-be-quantized object, the sensitivity of each hierarchical quantization strategy of each network layer can be determined, the quantization strategy of each network layer can be determined based on the sensitivity of each hierarchical quantization strategy, and the third quantization strategy corresponding to the first fallback solution can be obtained based on the quantization strategy of each network layer.

[0022] In the above implementation manner, when the optimal third quantization strategy under each fallback solution is determined, the quantization strategy of each network layer needs to be determined in a more fine-grained manner. If the traditional quantization search is used to determine the quantization strategy, there is still a problem of large search space. The sensitivity is used to determine the quantization strategy of each network layer, so that the efficiency of quantization fallback is improved. Compared with the solution of searching in the vast search space of the entire multi-network layer, multiple to-be-quantized objects and multiple data types, the optimal quantization strategy can be obtained more efficiently and with higher quality, thereby improving the quantization efficiency of the model.

[0023] In a possible implementation, the to-be-quantized objects include a linear transformation to-be-quantized object, a key-value cache (KV cache) to-be-quantized object, a set communication to-be-quantized object, and an attention calculation to-be-quantized object, where the linear transformation to-be-quantized object includes an operator performing linear transformation on data, an activation value parameter, and a weight parameter, the KV cache to-be-quantized object includes K and V vectors in the KV cache, and the attention calculation to-be-quantized object includes an operator of matrix multiplication calculation in an attention layer, and the set communication to-be-quantized object includes communication data in set communication.

[0024] The implementation maximizes the quantization degree of the model by simultaneously quantizing the plurality of to-be-quantized objects, and develops the potential of overall quantization of the model. Meanwhile, the best first quantization scheme is selected according to the sensitivity of the entire quantization strategy, which can avoid the problem that the model accuracy exceeds the controllable range due to the cumulative accuracy caused by quantization of the plurality of to-be-quantized objects.

[0025] In a possible implementation, the method further includes the following steps: in the case that the rollback evaluation results of each rollback scheme are all unqualified, analyzing the rollback evaluation results of each rollback scheme to obtain an analysis result, the analysis result including to-be-quantized objects and network layer information whose influence on quantization error is greater than a threshold, and sending a prompt message of manual rollback to a client held by a user, the prompt message including one or more of the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and the analysis result.

[0026] In the specific implementation, when prompting the user to manually rollback, the prompt message can include the optimal quantization strategy under each rollback scheme and the quantization evaluation result. Although the evaluation results of these quantization strategies are unqualified, they can be used as a reference for the user, so that the user can more quickly obtain the quantization strategy of manual rollback. Further, the quantization evaluation result of the optimal quantization strategy under each rollback scheme can be analyzed, and then the analysis result is displayed to the user, for example, the mean square error of each quantization strategy is displayed, so that the user can determine which to-be-quantized objects and which network layers have larger errors, and the user can modify the quantization data type of the to-be-quantized object to obtain a second quantization strategy.

[0027] In the above implementation, if the model accuracy required by the user is high and the performance of the model after quantization rollback cannot meet the user's requirement, the user can be prompted to obtain a custom quantization strategy. In order to facilitate the user to determine the quantization parameters, the network layers and parameters that have a great impact on performance can be identified to the user according to the previous quantization evaluation result and the rollback evaluation result, so as to facilitate the user to customize the quantization strategy and further improve the user experience.

[0028] In a second aspect, a model quantization system is provided, which includes: an acquisition unit configured to acquire one or more to-be-quantized objects of a to-be-quantized model and at least one data type that each to-be-quantized object can be quantized into; a combination unit configured to determine a plurality of quantization strategies based on the at least one data type that each to-be-quantized object can be quantized into, wherein each quantization strategy includes a quantization data type of each to-be-quantized object, and the quantization data type is one of the at least one data type that the to-be-quantized object can be quantized into; a determination unit configured to determine a sensitivity of each quantization strategy, wherein the sensitivity is used to indicate a degree of accuracy loss of the to-be-quantized model after the to-be-quantized model is quantized using the quantization strategy; and the determination unit is configured to acquire a first quantization strategy from the plurality of quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize the one or more to-be-quantized objects.

[0029] The system described in the second aspect is implemented, which freely combines the to-be-quantized objects of the to-be-quantized model and the data types that can be quantized, obtains a plurality of different quantization strategies, the data types corresponding to the to-be-quantized objects are different in each quantization strategy, then determines the sensitivities of the different quantization strategies based on a sensitivity algorithm, the sensitivity is used to indicate the influence of the quantization strategy on the accuracy of the to-be-quantized model, and the first quantization strategy is determined according to the sensitivity of each quantization strategy, so that the entire quantization process does not need to be quantized, thereby solving the problem of low quantization efficiency caused by long time consumption of quantization search.

[0030] In a possible implementation, the determination unit is configured to determine the sensitivity of each to-be-quantized object in each quantization strategy, and obtain the sensitivity of each quantization strategy based on the sensitivity of each to-be-quantized object, wherein the sensitivity of each to-be-quantized object is obtained based on one or more parameters of an average trace of a hessian matrix of the to-be-quantized object and a loss value before and after quantization of the to-be-quantized object.

[0031] In a possible implementation, the system further includes an evaluation unit and a fallback unit, the evaluation unit is configured to perform quantization evaluation on the first quantization strategy to obtain a quantization evaluation result, and the fallback unit is configured to perform quantization fallback on the first quantization strategy to modify the quantization data type of the to-be-quantized object in the first quantization strategy and obtain a second quantization strategy in a case where the quantization evaluation result is unqualified, and the second quantization strategy is used to quantize the one or more to-be-quantized objects.

[0032] In a possible implementation, the system further includes a quantization unit, the quantization unit is configured to quantize the to-be-quantized model using the first quantization strategy or the second quantization strategy to obtain a quantized model, and the quantization unit is configured to send the quantized model to an inference system.

[0033] In a possible implementation, the to-be-quantized model includes a plurality of network layers, in the first quantization strategy, quantization data types of a same to-be-quantized object in each network layer are same, and in the second quantization strategy, quantization data types of the same to-be-quantized object in each network layer are same or different, and a quantity of to-be-quantized objects in the second quantization strategy is less than that in the first quantization strategy.

[0034] In a possible implementation, the fallback unit is configured to obtain a plurality of fallback schemes, each of which includes at least one to-be-quantized object, and different fallback schemes include different quantities of to-be-quantized objects, determine quantization data types of each to-be-quantized object in a first fallback scheme in the plurality of fallback schemes, obtain a third quantization strategy corresponding to the first fallback scheme, perform quantization evaluation on the third quantization strategy to obtain a fallback evaluation result, determine the third quantization strategy as the second quantization strategy in a case where the fallback evaluation result is qualified, and determine quantization data types of each to-be-quantized object in a second fallback scheme in a case where the fallback evaluation result is unqualified, and obtain a fourth quantization strategy corresponding to the second fallback scheme, where the quantity of to-be-quantized objects in the first fallback scheme is not less than that in the second fallback scheme.

[0035] In a possible implementation, the fallback unit is configured to obtain, based on to-be-quantized objects included in the first fallback scheme and data types that can be quantized for each to-be-quantized object, at least one hierarchical quantization strategy of each network layer under the first fallback scheme, determine a sensitivity of each hierarchical quantization strategy of each network layer, determine a quantization strategy of each network layer based on the sensitivity of each hierarchical quantization strategy, and obtain a third quantization strategy corresponding to the first fallback scheme based on the quantization strategy of each network layer.

[0036] In a possible implementation, the to-be-quantized object includes a linear transformation to-be-quantized object, a key-value cache (KV cache) to-be-quantized object, a collective communication to-be-quantized object, and an attention calculation to-be-quantized object, the linear transformation to-be-quantized object includes an operator performing linear transformation on data, an activation value parameter, and a weight parameter, the KV cache to-be-quantized object includes a K vector and a V vector in the KV cache, the attention calculation to-be-quantized object includes an operator of matrix multiplication calculation in an attention layer, and the collective communication to-be-quantized object includes communication data in collective communication.

[0037] In a possible implementation, the system further includes a prompting unit, configured to analyze the rollback evaluation result corresponding to each rollback scheme in a case where the rollback evaluation result of each rollback scheme is all unqualified, to obtain an analysis result, the analysis result including the to-be-quantized object and the network layer information whose influence on the quantization error is greater than a threshold, and the prompting unit is configured to send a prompt message of manual rollback to the client, the prompt message including one or more of the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and the analysis result.

[0038] In a third aspect, a computing device is provided, including a processor and a memory, the memory being configured to store instructions, and the processor being configured to execute the instructions to enable the computing device to implement the method described in the first aspect.

[0039] In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing instructions, and the instructions being executed by a computing device or a computing device cluster to implement the method described in the first aspect.

[0040] In a fifth aspect, a computing device cluster is provided, including at least one computing device, each of the at least one computing device including a processor and a memory, and the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to enable the computing device cluster to implement the method described in the first aspect.

[0041] In a sixth aspect, a computer program product including instructions is provided, the instructions being executable on a computing device or being stored in any available medium or software or program product, and enabling the computing device or the computing device cluster to execute the method described in the first aspect when the computer program product is executed on the computing device or the computing device cluster.

[0042] On the basis of the implementation manners of the aspects described above, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0043] FIG. 1 is an architecture diagram of a model quantization system provided by the present application;

[0044] FIG. 2 is an example diagram of a model quantization system provided by the present application deployed in a cloud environment;

[0045] FIG. 3 is a step flowchart of a method for determining a quantization strategy provided by the present application;

[0046] FIG. 4 is an example diagram of a to-be-quantized object in a method for determining a quantization strategy provided by the present application;

[0047] FIG. 5 is a structural schematic diagram of a model quantization system provided by the present application;

[0048] FIG. 6 is a structural schematic diagram of a computing device provided by the present application;

[0049] FIG. 7 is an example diagram of a computing device cluster provided by the present application;

[0050] FIG. 8 is a structural schematic diagram of another computing device cluster provided by the present application. DETAILED DESCRIPTION

[0051] First, the application scenarios involved in the present application are explained and described.

[0052] With the continuous development of technology, the size of large language models (LLMs) is becoming larger and larger, and the inference latency of large language models is also becoming longer and longer. In order to reduce the storage overhead and inference latency of the model, model quantization technology has emerged. Model quantization refers to the process of converting high-precision parameters in a large language model into low-precision parameters, or converting high-precision calculations into low-precision calculations. Changing the precision of parameters can be referred to as parameter quantization, that is, converting the parameters (such as weights) to be quantized in the model from a high-precision data type to a low-precision data type, such as from a floating-point type to an integer type, thereby reducing the storage space occupied by the model. Changing the calculation precision can be referred to as operator quantization, that is, converting the operators (such as convolution, activation function, matrix multiplication, etc. calculation operations) to be quantized in the model from high-precision calculation to low-precision calculation, such as from floating-point operation to integer operation. Unlike parameter quantization, operator quantization focuses on the entire calculation process, not on a specific parameter itself. Operator quantization can quantize multiple parameters involved in the calculation, such as weights, input data, output data, and activation values. The present application does not make specific limitations. Simply put, parameter quantization is mainly used to reduce the model size, and operator quantization is mainly used to speed up the operation and reduce the computing resources. The focus of the two is different. By reducing the precision of the model parameters or calculations, the storage requirements and computational complexity of the large language model can be reduced, thereby reducing the storage overhead and inference latency of the large language model.

[0053] Model quantization technology changes the precision of model parameters and calculations, so it also brings precision loss to large language models. Different large language models have different precision loss performances, so each large language model needs to determine a targeted quantization strategy to ensure that the performance loss after model quantization is within an acceptable range. Usually, before model quantization, a search algorithm is used to determine the optimal quantization strategy for the large language model, including the parameters that need to be quantized and the quantization method of the parameters, and then the corresponding quantization operation is performed.

[0054] However, the large language model has a large size and a large number of parameters, so that the search space of the search algorithm is very large when the large language model is quantized, and the search time increases exponentially with the size of the search space, resulting in low efficiency of the large language model quantization.

[0055] Meanwhile, the inference process of the large language model is relatively complex and has many different calculation stages, and the objects to be quantized in each calculation stage are different, so the precision loss caused by quantization in each calculation stage is also different. If each calculation stage is quantized, the precision loss will be superimposed, resulting in a loss of the final model quantization beyond the acceptable range of the user. Therefore, the current large language model quantization will only select one object to be quantized, such as only quantizing the matrix multiplication (matmul) operator or only quantizing the data stored in the KV cache, and other calculation stages still maintain high-precision calculation, so that the quantization precision can be controlled. However, this will result in some parameters or calculations that can reduce precision not being quantized, limiting the overall quantization potential of the large language model, and the model quantization degree cannot meet the user's demand.

[0056] In summary, in the current large language model quantization process, due to the large size of the model, the search space of the quantization strategy is too large, the search time is long, and the large language model quantization efficiency is low. Moreover, the large language model can usually only be quantized for one calculation stage, otherwise the precision loss will be superimposed. However, quantizing only one calculation stage limits the overall quantization potential of the large language model, and the model quantization degree is low. Therefore, the current large language model has the problems of low quantization efficiency and low quantization degree in the model quantization process, making it difficult for the model quantization to meet the user's usage demand.

[0057] In order to solve the problem of large search space and long search time in the model quantization process, which leads to low model quantization efficiency, the application provides a method for determining a quantization strategy. The method obtains a plurality of different quantization strategies by freely combining the to-be-quantized objects supported by the to-be-quantized model and the quantizable data types, and then determines the sensitivity of different quantization strategies based on a sensitivity algorithm. The sensitivity is used to indicate the influence of the quantization strategy on the accuracy of the to-be-quantized model. The first quantization strategy is determined according to the sensitivity of each quantization strategy. Then, the first quantization strategy is quantized and evaluated. If the evaluation is qualified, the first quantization strategy is determined as the quantization strategy of the to-be-quantized model, so that the entire quantization process does not need to be quantized and searched, thereby solving the problem of low quantization efficiency caused by long quantization search time in the model quantization process. If the evaluation is not qualified, the quantization is rolled back based on the first quantization strategy, the accuracy of part of the to-be-quantized objects in the first quantization strategy is improved, and a second quantization strategy that passes the evaluation is obtained. Compared with the traditional quantization search method based on a large number of parameters, the method of determining the first quantization strategy and then rolling back can reduce the search space, thereby solving the problem of large search space and long search time in the model quantization process, which leads to low model quantization efficiency.

[0058] FIG. 1 is an architecture diagram of a model quantization system provided by the application. As shown in FIG. 1, the architecture includes a client 100, a model quantization system 200, and an inference system 300. The client 100, the model quantization system 200, and the inference system 300 are communicatively connected, which can be wired network connection or wireless network connection. The wired network connection includes Ethernet, occupied line, cable, etc. The wireless network connection includes wireless fidelity (Wi-Fi), cellular network, etc. It can also include internet, local area network (LAN), etc. that support wired network connection and wireless network connection at the same time. Of course, it can also include other network types, which are not limited in the application. The number of clients 100 and inference systems 300 can be one or more. FIG. 1 takes one client and one inference system as an example for illustration, which is not limited in the application.

[0059] The client 100 is deployed on a terminal device or a computing device for realizing human-computer interaction. The terminal device includes a personal computer, a smart phone, a wearable device, a palm-held processing device, a tablet computer, a mobile notebook, an augmented reality (AR) device, a virtual reality (VR) device, a smart conference device, and the like, which are not specifically limited herein. The computing device can be a bare metal server (BMS), a virtual machine, or a container. The BMS refers to a general-purpose physical server, for example, an ARM server or an X86 server; the virtual machine refers to a complete computer system that is simulated by software, runs in a completely isolated environment, and has complete hardware system functions. The work that can be completed in a physical computer can also be realized in the virtual machine. When a virtual machine is created in a computing device, part of the hard disk and memory capacity of the physical machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent basic input / output system (BIOS), a hard disk, and an operating system, and can be operated like a physical machine; the container is a portable software unit that can combine an application and all its dependencies into a software package, which is not limited by the underlying host operating system, so that a complex environment no longer needs to be built, and the process of application development to deployment is simplified.

[0060] The model quantization system 200 can be deployed on a computing device or a computing device cluster composed of multiple computing devices, the description of which can refer to the above description, which is not repeated here. It can also be deployed on a chip or a chip system composed of multiple chips, which can include a processor implemented by a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). Among them, the above-mentioned PLD can be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), or any combination thereof, which is not limited in the present application. It can also include an acceleration card that has good performance when performing machine learning tasks and can improve the speed and efficiency of neural network training, such as a graphics processing unit (GPU), a neural network processing module (NPU), a data processing unit (DPU), a tensor processing unit (TPU), etc.

[0061] The inference system 300 can be deployed on the terminal device, the computing device or the computing device cluster described above, and can also be deployed on an inference card, an edge device, etc., which is not limited in the present application. Among them, the inference card is a hardware specially designed to accelerate the AI model inference process, and the edge device is a hardware device close to the data source or the data user.

[0062] Optionally, the client 100 and the model quantization system 200 can be deployed on the same computing device, or the client 100 and the model quantization system 200 can be deployed on different computing devices in the same computing device cluster, or the client 100 and the model quantization system 200 can be deployed on different computing devices in different computing device clusters, which is not limited in the present application.

[0063] Optionally, the inference system 300 and the model quantization system 200 can be deployed on the same computing device, or the inference system 300 and the model quantization system 200 can be deployed on different computing devices in the same computing device cluster, or the inference system 300 and the model quantization system 200 can be deployed on different computing devices in different computing device clusters, and the present application does not make specific limitations.

[0064] Optionally, the client 100 can be a terminal device or a software or application running on a computing device controlled by a user, such as a personal computer (PC) client, a web client based on browser access, an application (APP) client running on a mobile terminal, or a console of a cloud platform, and the present application does not make specific limitations. The user holding the client 100 can be a developer of an AI model, such as a programmer or an AI engineer, and the present application does not make specific limitations.

[0065] Optionally, the client 100 can be a model quantization tool, which is specifically used to implement model quantization. The user can use the model quantization tool to quantize the model before deploying the model in a production environment (that is, the inference system 300), reduce the precision of parameters or calculations in the model, and thus reduce the memory requirement and the calculation resource consumption of the model, and then deploy the model in the production environment, so that the model can run on more lightweight devices such as smartphones and edge devices while maintaining performance. The model quantization tool can implement model quantization by using the method for determining a quantization strategy provided in the present application.

[0066] Optionally, the client 100 can be a client of a model service platform or a machine learning framework for processing machine learning related tasks. Such a client provides a series of tools and services for the whole process of definition, training, deployment, inference, and management of a machine learning model, and such a client includes the model quantization function provided in the present application to implement the quantization compression of the model.

[0067] Optionally, the client 100 can also be a client of a cloud platform for users to purchase and rent various cloud services, and the method for determining a quantization strategy provided in the present application can be one of the cloud services, and the user can purchase the cloud service alone; or the cloud platform provides an integrated service to the user, and the method for determining a quantization strategy provided in the present application can be a sub-service in the integrated cloud service, for example, the integrated cloud service is an AI cloud service, and the method for determining a quantization strategy provided in the present application can be a sub-service of the AI cloud service. After the model is trained and before it is deployed in a production environment, the sub-service can be used to quantize and compress the model.

[0068] The possible deployment modes of the client 100, the model quantization system 200, and the inference system 300 are described in detail above, and in actual deployment, flexible deployment can be performed in combination with specific application scenarios and business requirements. The actual deployment modes of the client 100, the model quantization system 200, and the inference system 300 are described below by way of example.

[0069] For example, the client 100, the model quantization system 200, and the inference system 300 can be deployed in an enterprise's internal network, the model quantization system 200 and the inference system 300 are deployed in a server or a server cluster purchased by the enterprise, and the client 100 is a remote desktop connection tool or a management console software of a server. The employees of the enterprise can remotely connect to the server or the server cluster where the model quantization system 200 is located through the client 100, use the model quantization system 200 to quantize and compress the trained model, and deploy the quantized and compressed model in the server or the server cluster where the inference system 300 is located.

[0070] For another example, the client 100, the model quantization system 200, and the inference system 300 can be deployed in a cloud environment. For example, FIG. 2 is an example diagram of a model quantization system deployed in a cloud environment according to the present application. As shown in FIG. 2, a user can initiate a purchase request for an AI cloud service through the client 100, and the AI cloud service includes a sub-service corresponding to the method for determining a quantization strategy provided in the present application. Specifically, after the client 100 sends the purchase request to the cloud platform, the cloud platform can provide the client 100 with the use right of the model quantization system 200 provided in the present application, so that the user can use the model quantization system 200 to quantize and compress the model through the client 100, and after obtaining the quantized and compressed lightweight model, the user can deploy the lightweight model in the inference system 300.

[0071] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, security resources, and the like, to meet the computing requirements of the model quantification system 200 under different scales and loads, and the computing resources can be dynamically scaled according to the use requirements of the model quantification system 200, to ensure stable operation of the model quantification system 200 and provide reliable model quantification services for users.

[0072] Optionally, the inference system 300 and the model quantification system 200 can be services provided by the same cloud platform. At this time, the inference system 300 and the model quantification system 200 are included in the cloud platform shown in FIG. 2. Alternatively, the inference system 300 and the model quantification system 200 can be services provided by different cloud platforms. At this time, the data communication between the inference system 300 and the model quantification system 200 can be realized through a hybrid cloud architecture, that is, the model quantification system 200 is deployed in data center A, the inference system 300 is deployed in data center B, and the data communication between the inference system 300 and the model quantification system 200 is realized through the respective cloud platforms of the data center A and the data center B.

[0073] It should be noted that in the example shown in FIG. 2, the model quantification system 200 is deployed and implemented in a cloud environment, and FIG. 2 is only an exemplary description of the model quantification system 200 involved in the present application, which is not specifically limited.

[0074] Optionally, the model quantification system 200 can also be deployed and implemented through multiple cloud environments, for example, the confirmation process of the first quantization strategy shown in FIG. 1 is implemented in a first cloud environment, the process of quantization evaluation and obtaining the quantization evaluation result of the first quantization strategy is implemented in a second cloud environment, the quantization rollback is implemented in a third cloud environment, and the final model quantization result is obtained in a fourth cloud environment. The above examples are used for illustration, and the present application is not specifically limited.

[0075] Optionally, the model quantification system 200 can also be deployed and implemented through multiple non-cloud environments, for example, the confirmation process of the first quantization strategy shown in FIG. 1 is implemented on an internal office server of enterprise A, the process of quantization evaluation and obtaining the quantization evaluation result of the first quantization strategy and the process of quantization rollback are implemented on an internal office server of enterprise B, and the final model quantization result is implemented on an internal office server of enterprise C. The above examples are used for illustration, and the present application is not specifically limited.

[0076] Optionally, the model quantization system 200 can also be implemented through multiple cloud environment and non-cloud environment deployment. For example, the confirmation process of the first quantization strategy shown in FIG. 1 is implemented on the internal office server of enterprise A, the process of obtaining quantization evaluation results by quantization evaluation of the first quantization strategy and the process of quantization rollback are implemented in the first cloud environment, and the final model quantization result is implemented on the internal office server of enterprise B. The above examples are used for illustration, and the application is not limited specifically.

[0077] It should be understood that the above describes some possible deployment forms of the model quantization system 200 in conjunction with specific scenarios, and other deployment forms can also exist in specific implementations. The application does not limit the deployment of the model quantization system 200. For ease of description, the following embodiments are described with reference to the system structure shown in FIG. 1.

[0078] It should be understood that the model quantized by the application can be any model in the AI field. As known from the foregoing, the more model parameters, the worse the quantization effect. The large language model is a model with a large scale and a large number of parameters. For ease of description, the following embodiments are described with reference to the large language model.

[0079] In the embodiments of the application, the model quantization system 200 can obtain a to-be-quantized model from the client 100, confirm that the to-be-quantized model supports at least one to-be-quantized object to be quantized, and each to-be-quantized object can be quantized at least one data type, generate multiple quantization strategies through permutation and combination, each quantization strategy includes the quantization data type of each to-be-quantized object, the quantization data type is one of the at least one data type, and then calculate the sensitivity corresponding to each quantization strategy. The sensitivity is used to indicate the degree of accuracy loss of the quantization strategy. Then, based on the sensitivity of each quantization strategy, a first quantization strategy with the lowest degree of accuracy loss is obtained, and the first quantization strategy is evaluated. In the case of a qualified evaluation result, the to-be-quantized model is quantized using the first quantization strategy to obtain a model quantization result; in the case of an unqualified evaluation result, the accuracy of the quantization data type of some to-be-quantized objects is improved by quantization rollback of the first quantization strategy until the evaluation result is qualified, a second quantization strategy is obtained, and then the to-be-quantized model is quantized using the second quantization strategy to obtain a model quantization result. Then the model quantization result is sent to the inference system 300 and enters the production environment.

[0080] For example, taking Figure 1 as an example, after the model quantization system 200 receives the model to be quantized sent by the client 100, it determines the various objects to be quantized in the model. Figure 1 uses four objects to be quantized as an example. Then, it determines at least one data type that each object can be quantized. Figure 1 illustrates this by showing that each object has two data types. Then, by arranging and combining these data types, two... 4 =16 permutations and combinations, which are 16 quantization strategies (Figure 1 shows two examples). Then, the sensitivity of each quantization strategy is calculated. Assuming that the lower the sensitivity, the smaller the accuracy loss, the scheme with the lowest sensitivity can be selected as the first quantization strategy. In the example of Figure 1, assuming that quantization strategy 2 is the first quantization strategy, quantization strategy 2 can be evaluated. If it is qualified, the model will be quantized according to the scheme to obtain the model quantization result. If it is not qualified, quantization strategy 2 will be rolled back. For example, if the accuracy of data type 1 is lower than that of data type 2, the data type of the object to be quantized, 4, can be changed from data type 2 to data type 1. Then, it is determined whether the model accuracy after this rollback meets the requirements. If it does not meet the requirements, the data type of the object to be quantized, 3, can be changed from data type 2 to data type 1. Then, it is determined whether the model accuracy after this rollback meets the requirements. This process is repeated until the model accuracy meets the requirements, and the final quantization strategy is obtained. Then, the model is quantized using this scheme to obtain the model quantization result.

[0081] Optionally, the object to be quantized refers to an object in the model that can be quantized. Here, "object to be quantized" is a category, and each object to be quantized can include one or more quantization parameters or quantization operators. Parameters in the model can be classified according to their function. Each network layer may include many parameters of the same type. Some types of parameters can be quantized, while others cannot. For example, some dynamic parameters, if quantized, would require dynamic adjustment, making quantization difficult; these types of parameters generally do not participate in quantization. Similarly, some specific activation functions, such as the ReLU function, may lead to information loss after quantization; these types of parameters generally do not participate in quantization. Small parameter values, such as those close to 0, may lead to a loss of numerical precision; these types of parameters generally do not participate in quantization. Of course, there are other parameters that cannot participate in quantization, which will not be listed here. It should be understood that the object to be quantized in this application refers to a certain type of parameter that can participate in quantization.

[0082] For example, the to-be-quantized objects can include a linear transformation to-be-quantized object, a KV cache to-be-quantized object, an attention calculation to-be-quantized object, and a set communication to-be-quantized object, wherein the linear transformation to-be-quantized object includes an operator performing linear transformation on data, and further includes an activation value parameter and a weight parameter participating in the linear transformation, the KV cache to-be-quantized object includes K and V vectors in the KV cache, the attention calculation to-be-quantized object includes an operator of matrix multiplication calculation in an attention layer, and the set communication to-be-quantized object includes communication data in set communication.

[0083] Optionally, the data type refers to a numerical format of data in the model, and the data type includes a float type and an integer type. The float type includes float16 (fp16), float32 (fp32), and the like, wherein fp16 represents a 16-bit floating-point number format, which is composed of 1-bit sign bit, 5-bit exponent bit, and 10-bit mantissa bit, and fp32 represents a 32-bit floating-point number format, which is composed of 1-bit sign bit, 8-bit exponent bit, and 23-bit mantissa bit. The larger the number is, the greater the numerical precision is, and the greater the demand for storage and computing resources is. The integer type includes int8, int4, and the like, wherein int8 represents an 8-bit integer format, and the represented range is -128 to 127, or 0 to 255 (unsigned), and int4 represents a 4-bit integer format, and the represented range is -8 to 7, or 0 to 15 (unsigned). It can be understood that the data precision of the float type is greater, and the demand for storage and computing resources is greater. The data precision of the integer type is smaller, and the demand for storage and computing resources is smaller.

[0084] Optionally, the data type that can be quantized by the to-be-quantized object includes a data type that can be used by the to-be-quantized object. Generally, in combination with a model structure, a hardware platform, a computing efficiency, and a task demand, different to-be-quantized objects can use different data types. For example, in a high-precision task scenario, such as medical image processing and financial prediction, a data type with a higher bit width, such as fp16 and fp32, is required. For some scenes that allow a certain error, such as object detection or image classification tasks, a data type with a lower bit width, such as int8 and int4, can be used. Different hardware platforms support different data types. For example, some CPUs and GPUs support int8 operation, while some TPUs can have special optimization for fp16. Some embedded devices or edge devices can only support a data type with a lower bit width, such as int8 or int4, due to limited resources. Therefore, the data type that can be used by the to-be-quantized object needs to be determined in combination with a specific scene.

[0085] Further, as known from the foregoing, quantization refers to converting a model from high precision to low precision, and thus the precision of the quantizable data type to which the to-be-quantized object can be quantized is lower than the current data type. For example, if the to-be-quantized object is currently in the fp16 floating-point number format, the quantizable data type can only be a data type with lower precision than fp16 (such as the integer type int8 or int4), but not a data type with higher precision than fp16 (such as fp32). The above examples are used for illustration, and the present application is not limited in this regard.

[0086] Optionally, the sensitivity of each quantization strategy can be obtained based on one or more sensitive parameters related to the precision of the model. Specifically, the one or more sensitive parameters can be calculated, and then the one or more sensitive parameters can be calculated by weighted summation or multiplication to determine the sensitivity. The sensitive parameters can include the average trace of the Hessian matrix of the to-be-quantized object, the loss function values of the to-be-quantized object before and after quantization, and more, which can be determined according to the actual business scenario, and the present application is not limited in this regard. The Hessian matrix is a matrix of second-order partial derivatives of a multivariate function, which is usually used to describe the local curvature of a function. The matrix can help determine the extreme situation of a function at a certain point and the properties of the extreme point, which can be a maximum value, a minimum value, or a saddle point. The average trace of the Hessian matrix refers to the sum of the diagonal elements of the matrix.

[0087] For example, for a linear transformation to-be-quantized object, the sensitivity can be obtained based on the product of the average trace of the Hessian matrix of the linear layer parameters and the loss function value, where the linear layer parameters include activation values and weights. For a KV cache to-be-quantized object, the sensitivity can be obtained based on the average trace of the Hessian matrix of the K vector and the V vector stored in the KV cache. For an attention calculation to-be-quantized object, the sensitivity can be obtained based on the average trace of the Hessian matrix of the Q vector, the K vector, and the V vector participating in attention calculation. For a set communication to-be-quantized object, the sensitivity can be obtained based on the average trace of the Hessian matrix of the to-be-communicated data participating in set communication. The above examples are used for illustration, and the present application is not limited in this regard.

[0088] It should be understood that the model quantization system 200 of the present application first determines the to-be-quantized objects in the model that can be quantized and the data types that each to-be-quantized object can be quantized, and then determines the quantization strategy of the model through permutation and combination, the quantization strategy including the quantized data type corresponding to each to-be-quantized object, the quantized data type being one of the above-mentioned quantizable data types, and then determining the sensitivity value of each quantization strategy. In this way, the precision is evaluated in units of quantization strategies, which can avoid the problem of accumulated precision leading to model precision exceeding the controllable range due to quantization of multiple to-be-quantized objects, so that the entire model can simultaneously quantize to-be-quantized objects in multiple calculation stages, the quantization degree of the model can be maximized, and the potential of overall model quantization can be developed. At the same time, the sensitivity is used to judge the precision loss of the quantization strategy to the model, and the model under each quantization strategy does not need to be quantized and evaluated using a test set, and the first quantization strategy can be quickly screened based on the sensitivity.

[0089] Moreover, if the performance of the model meets the standard after the first quantization strategy determined based on the sensitivity value is evaluated by quantization, the first quantization strategy can be directly used as the final solution to quantize the model, so that the entire quantization process does not need to be searched for quantization, and the quantization evaluation only needs to be performed once, thereby solving the problem of low quantization efficiency of the model quantization due to long time consumption of the quantization search.

[0090] If the model does not meet the standard, a second quantization strategy can be searched for by quantization rollback, which can reduce the search space compared to the traditional way of searching for the corresponding quantized data type for a large number of parameters one by one, so as to improve the efficiency of quantization of the large language model.

[0091] For example, assuming that the model has 10,000 parameters, each parameter can have three data types, and the search space is 3 10000 , the search space is very large and the calculation cost is extremely high. Using the technical solution of the present application, assuming that the first quantization strategy is that all parameters use data type 1, and the search space is maximally 3 10000 , that is, each parameter needs to be adjusted, which is the worst case. If only one parameter needs to be adjusted, it is the best case, and the search space of the rollback is 3 1000 , which is much smaller than the search space of the traditional technology. Therefore, using the technical solution of the present application can significantly reduce the number of quantization searches, and if the performance of the first quantization strategy meets the standard, even no quantization search is needed, which greatly improves the efficiency of model quantization.

[0092] Optionally, the model quantization system 200 can pre-set M fallback schemes, each of which includes different quantization data types corresponding to the to-be-quantized objects, then determine the optimal quantization strategy under the fallback scheme according to the to-be-quantized objects and the quantization data types in the fallback scheme, and then perform quantization evaluation, if it is qualified, it is used as the second quantization strategy, if it is not qualified, the next to-be-quantized object and the quantization data type in the next fallback scheme are determined, the optimal quantization strategy under the next fallback scheme is determined, and so on, until the second quantization strategy with a qualified quantization evaluation result is obtained. In this way, only M times of evaluation are needed to obtain the second quantization strategy, which can improve the efficiency of quantization fallback.

[0093] For example, scheme one includes linear transformation to-be-quantized objects and KV cache to-be-quantized objects, which means that only the two to-be-quantized objects of the model are quantized, and other to-be-quantized objects are not quantized, scheme two includes linear transformation to-be-quantized objects, KV cache to-be-quantized objects and set communication objects, which means that the three to-be-quantized objects of the model are quantized, and so on. Here, it is not necessary to enumerate all the examples.

[0094] Further, the model quantization system 200 can determine the search order of the quantization strategy according to the number of to-be-quantized objects in the M fallback schemes. The more to-be-quantized objects in the fallback scheme, the earlier the search order. In other words, when searching, the number of to-be-quantized objects participating in the search is gradually reduced, and the number of to-be-quantized objects for fallback is gradually increased. For example, if there are five to-be-quantized objects, first search according to four to-be-quantized objects, that is, fallback one to-be-quantized object, then search according to three to-be-quantized objects, that is, fallback two to-be-quantized objects, then search according to two to-be-quantized objects, that is, fallback three to-be-quantized objects, and so on. In this way, the number of to-be-quantized objects participating in quantization is gradually reduced, which can avoid large-scale fallback and improve the efficiency of fallback.

[0095] For example, in the example shown in FIG. 1, when quantization fallback is performed on the quantization strategy 2, first search according to to-be-quantized objects 1, 2 and 3, perform quantization evaluation on the searched quantization strategy, if it is qualified, the quantization strategy is the final quantization strategy, if it is not qualified, search according to to-be-quantized objects 1, 2 and 4, perform quantization evaluation on the searched quantization strategy, and so on. If all the fallback schemes containing three to-be-quantized objects do not search a qualified quantization strategy, start searching the fallback schemes containing two to-be-quantized objects, and so on, until a quantization evaluation qualified quantization strategy is searched. The above examples are used for illustration, and the present application is not limited in this way.

[0096] Further, when searching for the optimal quantization strategy according to the fallback scheme, the model quantization system can also use the above sensitivity calculation method to calculate the optimal quantization strategy. If the to-be-quantized object includes parameters of multiple network layers, the model quantization system can calculate the quantization data type of the parameters corresponding to each network layer according to the network layer as the granularity, to obtain a more refined quantization strategy. That is, in the first quantization strategy, all parameters included in a to-be-quantized object use a unified quantization data type, for example, the activation values and weights of all linear layers are quantized to w8a8. When quantization fallback is performed, the activation values and weights of each linear layer can be determined to have a quantization data type, for example, the first network layer is w8a8, the second network layer is w4a8, the third network layer is w4a16, and so on. The present application is not limited in this regard.

[0097] It should be understood that, when quantization fallback is performed, the quantization strategy is searched in a more fine-grained manner, and the fine-grained search is completed according to a certain fallback order. This not only can obtain a more refined and accurate quantization strategy, but also can improve the search efficiency. Compared with the scheme of directly searching in the massive search space of multiple network layers, multiple to-be-quantized objects, and multiple data types, the technical scheme of the present application can more efficiently and accurately obtain the optimal quantization strategy, thereby improving the quantization efficiency of the model.

[0098] It should be understood that, after the second quantization strategy is determined, the model quantization system 200 can feed back the scheme to the client 100. The user can select whether to use the scheme to quantize the model, or the model quantization system 200 can directly use the second quantization strategy to quantize the model, obtain the quantized model data, and then export the quantization result to the client 100. The user can select whether to deploy the model data in the inference system 300, or the model quantization system 200 can use the second quantization strategy to quantize the model, obtain the quantized model data, and then automatically deploy the model data in the inference system 300, to simplify the user operation process. The specific implementation can be determined according to the actual business requirements, and the present application is not limited in this regard.

[0099] In conclusion, the application provides a model quantization system. The system freely combines the to-be-quantized objects supported by the to-be-quantized model and the quantizable data types, obtains multiple different quantization strategies, determines the sensitivity of different quantization strategies based on a sensitivity algorithm, the sensitivity is used to indicate the influence of the quantization strategy on the precision of the to-be-quantized model, determines a first quantization strategy according to the sensitivity of each quantization strategy, and then performs quantization evaluation on the first quantization strategy. If the evaluation is qualified, the first quantization strategy is determined as the quantization strategy of the to-be-quantized model, so that the entire quantization process does not need to perform quantization search, thereby solving the problem of low quantization efficiency caused by long time consumption of quantization search in model quantization. If the evaluation is unqualified, quantization rollback is performed based on the first quantization strategy, the precision of part of the to-be-quantized objects in the first quantization strategy is improved, until a second quantization strategy that passes the evaluation is obtained. Compared with the traditional quantization search mode based on large-scale parameters, the mode of determining the first quantization strategy and then rolling back can reduce the search space, thereby solving the problem of large search space and long search time when searching for a quantization strategy in the model quantization process, which leads to low model quantization efficiency.

[0100] At the same time, evaluating the precision in units of quantization strategies can avoid the problem of model precision exceeding the controllable range caused by the accumulation of the precision of multiple to-be-quantized objects, so that the entire model can quantize the to-be-quantized objects in multiple calculation stages at the same time, and the quantization degree of the model can be maximized, thereby developing the potential of the overall model quantization.

[0101] The above describes the model quantization system provided by the application in combination with FIG. 1 and FIG. 2. The following describes the method for determining a quantization strategy provided by the application in combination with FIG. 3-FIG. 4. The method for determining a quantization strategy can be applied to the model quantization system 200 shown in FIG. 1 and FIG. 2.

[0102] FIG. 3 is a method for determining a quantization strategy provided by the application. As shown in FIG. 3, the method can include the following steps:

[0103] S310: The client 100 sends a model quantization request to the model quantization system 200.

[0104] In specific implementation, the model quantization request can include information of the to-be-quantized model, which is used for the model quantization system 200 to obtain model data of the to-be-quantized model. The information can include a model file and a configuration file of the to-be-quantized model, or a reading mode of the model file and the configuration file, or an identifier of the model file and the configuration file, etc. The information can also include other information that can be used by the model quantization system 200 to obtain the to-be-quantized model, which is not exemplified one by one here.

[0105] Optionally, the model quantization request can further include more content, such as quantization configuration information including calibration data set, precision loss tolerance, etc., running environment information including relevant information of the hardware device where the inference system to be deployed after model quantization, user-defined options including parameter information required for the quantization report, sensitivity calculation method, etc., and more content, which are not limited in the present application.

[0106] S320: The model quantization system 200 obtains the to-be-quantized object of the to-be-quantized model, and the data type that the to-be-quantized object can be quantized.

[0107] In specific implementation, the model quantization system 200 can obtain the model data of the to-be-quantized model according to the model quantization request, determine at least one to-be-quantized object supported by the to-be-quantized model for quantization and at least one data type that each to-be-quantized object can be quantized based on the model data. The data type that each to-be-quantized object can be quantized can be the same or different. For example, the model quantization system 200 can determine that the to-be-quantized object supported by the to-be-quantized model for quantization includes to-be-quantized objects A-C according to the model data, wherein the data type that the to-be-quantized object A can be quantized includes types 1-3, the data type that the to-be-quantized object B can be quantized includes types 1-5, and the data type that the to-be-quantized object C can be quantized includes types 1-3. The above examples are used for illustration, and the present application is not limited specifically.

[0108] Optionally, the to-be-quantized object refers to an object in the model that can be quantized. The to-be-quantized object here is a category, and each to-be-quantized object can include one or more quantization parameters or quantization operators. For example, the to-be-quantized object can include a linear transformation to-be-quantized object, a KV cache to-be-quantized object, an attention calculation to-be-quantized object, and a set communication to-be-quantized object. The linear transformation to-be-quantized object includes an operator for performing linear transformation on data, and also includes an activation value parameter and a weight parameter participating in the linear transformation. The KV cache to-be-quantized object includes K vector and V vector in the KV cache. The attention calculation to-be-quantized object includes an operator for matrix multiplication calculation in the attention layer. The set communication to-be-quantized object includes communication data during set communication. Set communication includes reduce scatter, all reduce, all gather, etc., which are not limited specifically in the present application.

[0109] It should be understood that different types of models can have different to-be-quantized objects. As known from the foregoing, the large language model has more calculation stages, and each calculation stage can have one or more to-be-quantized objects. The to-be-quantized objects of the large language model are introduced below with reference to FIG. 4.

[0110] FIG. 4 is an example diagram of an object to be quantized in a method for determining a quantization strategy provided by the present application, wherein FIG. 4 takes a large language model with a transformer architecture as an example to describe an object to be quantized that can exist in a network layer of this type of model.

[0111] As shown in FIG. 4, the input data is first normalized by a normalization layer to obtain a normalization result, reducing the difference in input distribution of different layers, which helps the stability and training efficiency of the model; then enters the attention mechanism layer, and the normalization result generates three vectors: K vector, V vector and Q vector through three parallel linear transformations (the normalization result is multiplied by three weight matrices obtained through training); then the K vector and the V vector are cached for subsequent repeated use, reducing the amount of calculation; then the Q vector, the K vector and the V vector are used to perform attention calculation to obtain an attention calculation result; the attention calculation result can be input into a fully connected layer to perform linear transformation (the attention calculation result is multiplied by a weight matrix, and an addition operation is performed with a bias vector) to generate a fully connected layer output result; then the fully connected layer output result is updated synchronously with the fully connected layer output result on other nodes through a collective communication algorithm such as the allreduce communication algorithm, to ensure that each node obtains consistent model parameters and obtains a collective communication result; based on the collective communication result, normalization is performed again to reduce the difference in input distribution of different layers, improving the stability and training efficiency of the model; then the input dimension is mapped to a higher dimension through linear transformation, also known as up projection (up) calculation; then a nonlinear activation function is used to process the output of the previous step, such as the gelu function; then the output data of the previous layer is mapped from a high dimension to the original or a lower dimension through linear transformation, also known as down projection (down) calculation; then the data on each node is synchronized through collective communication such as the allreduce communication to obtain the final output result.

[0112] The step-by-step process in FIG. 4 is described in detail above, and the object to be quantized that can be quantized is described below. FIG. 4 uses dark shading to mark the object to be quantized.

[0113] The linear transformation to-be-quantized objects include operators and parameters related to the linear transformation. In FIG. 4, there are four linear transformation to-be-quantized objects, which are respectively: a Q vector, a K vector, and a V vector generated by linear transformation, full connection layer calculation by linear transformation, up calculation by linear transformation, and down calculation by linear transformation. The four calculations can be matrix multiplication calculation (matmul). When quantizing such to-be-quantized objects, the data types of part or all of the parameters participating in the matrix multiplication calculation can be changed to reduce the data precision. It should be understood that the number of to-be-quantized objects of the same type can be one or more, such as the four linear transformation to-be-quantized objects in the example of FIG. 4.

[0114] The KV cache to-be-quantized object includes a K vector and a V vector in the KV cache. In FIG. 4, there is one KV cache to-be-quantized object. When quantizing such to-be-quantized objects, the data types of the accessed K vector and V vector can be changed to reduce the data precision.

[0115] The set communication to-be-quantized object includes communication data during set communication. In FIG. 4, there are two set communication to-be-quantized objects. When quantizing such to-be-quantized objects, the data types of the communication data can be changed to reduce the data precision.

[0116] The attention calculation to-be-quantized object includes operators and parameters related to the attention calculation process, that is, operators and parameters participating in the process of attention calculation of the Q vector, the K vector, and the V vector. In FIG. 4, there is one attention calculation to-be-quantized object. Although there is only one place in FIG. 4, there can be multiple matrix multiplication calculations in the actual attention calculation process, which can be determined according to the used attention calculation method. For example, in the flash attention algorithm, one matrix multiplication calculation is the matrix multiplication of the Q vector and the K vector, which is used to calculate the attention score, and the other matrix multiplication calculation is the matrix multiplication of the attention score and the V. When quantizing such to-be-quantized objects, the data types of part or all of the parameters participating in the matrix multiplication calculation can be changed to reduce the data precision.

[0117] Therefore, the scenario shown in the example of FIG. 4 includes eight to-be-quantized objects, one KV cache to-be-quantized object, one attention calculation to-be-quantized object, two set communication to-be-quantized objects, and four linear transformation to-be-quantized objects. It should be understood that FIG. 4 is an example of the to-be-quantized objects provided in the present application, and other structures of models can have more or fewer to-be-quantized objects, which are not specifically limited in the present application. Moreover, FIG. 4 is an example of to-be-quantized objects included in one network layer. A to-be-quantized model can include M network layers as shown in FIG. 4, and the to-be-quantized model can include at least 8M to-be-quantized objects.

[0118] Generally, the calculation, communication and storage related steps of the model can be quantized, and there are corresponding objects to be quantized. The method for determining a quantization strategy provided in the present application can quantize multiple objects to be quantized of the model at the same time, greatly improve the quantization degree of the model, and avoid the precision loss superimposed by the simultaneous quantization of multiple objects to be quantized through a series of algorithms, so that the model is quantized to the maximum extent while ensuring that the model precision loss is within a controllable range, meeting the user's usage requirements.

[0119] Optionally, the data type refers to the numerical format of the data in the model, and the data type includes a floating point type (float) and an integer type (linteger). The data type of the floating point type includes float16 (fp16), float32 (fp32) and the like, wherein fp16 represents a 16-bit floating point number format, which is composed of 1 bit of sign bit, 5 bits of exponent bit and 10 bits of mantissa bit, and fp32 represents a 32-bit floating point number format, which is composed of 1 bit of sign bit, 8 bits of exponent bit and 23 bits of mantissa bit. The larger the number is, the greater the numerical precision is, and the greater the demand for storage and computing resources is. The data type of the integer type includes int8, int4 and the like, wherein int8 represents an 8-bit integer format, and the range represented is -128 to 127, or 0 to 255 (unsigned), and int4 represents a 4-bit integer format, and the range represented is -8 to 7, or 0 to 15 (unsigned). It can be understood that the data precision of the floating point type is greater, and the demand for storage and computing resources is greater. The data precision of the integer type is smaller, and the demand for storage and computing resources is smaller.

[0120] For example, for the linear transformation object to be quantized, the corresponding data types can include w16a16, w8a16, w8a8 and w4a16, wherein w represents a weight, a represents an activation value, w8a8 means that the weight and the activation value are quantized to 8 bits in the matrix multiplication process, so that the data type of the activation value is int8 and the data type of the weight is int8. Similarly, w8a16 means that the weight is quantized to 8 bits, so that the data type of the weight is int8 and the data type of the activation value is fp16. The same applies to the other cases, which are not described one by one here. For the KV cache object to be quantized, the corresponding data types can include 16 bits, 8 bits and 4 bits. For the set communication object to be quantized, the corresponding data types can include 16 bits and 8 bits. For the attention calculation object to be quantized, the corresponding data types can include 16 bits and 8 bits. The above examples are used for illustration, and the present application is not limited in this regard.

[0121] S330: The model quantization system 200 combines multiple quantization strategies based on the objects to be quantized and the data types.

[0122] In specific implementation, each quantization strategy includes a quantization data type corresponding to each to-be-quantized object, and there is at least one to-be-quantized object in any two quantization strategies, which has different corresponding quantization data types in the two quantization strategies. That is, the quantization data type corresponding to each to-be-quantized object can be freely combined to form a plurality of different quantization strategies. For example, the to-be-quantized objects include to-be-quantized object 1 and to-be-quantized object 2, wherein the quantizable data types of to-be-quantized object 1 include A and B, and the quantizable data types of to-be-quantized object 2 include A and B. After free combination, four quantization strategies can be obtained. In quantization strategy 1, the quantization data types of to-be-quantized object 1 and to-be-quantized object 2 are A and A respectively, in quantization strategy 2, the quantization data types of to-be-quantized object 1 and to-be-quantized object 2 are A and B respectively, in quantization strategy 3, the quantization data types of to-be-quantized object 1 and to-be-quantized object 2 are B and A respectively, and in quantization strategy 4, the quantization data types of to-be-quantized object 1 and to-be-quantized object 2 are B and B respectively. The above examples are used for illustration, and the present application is not limited specifically.

[0123] Further, in each quantization strategy, the quantization data types corresponding to the same to-be-quantized objects in each network layer are the same. For example, in the example of FIG. 4, one network layer includes eight to-be-quantized objects, so the quantization data types of the KV cache to-be-quantized objects of each network layer are the same, the quantization data types of the attention calculation to-be-quantized objects of each network layer are the same, and so on.

[0124] For example, in the example shown in FIG. 4, one network layer includes eight to-be-quantized objects, the quantizable data types of the linear transformation to-be-quantized objects include w16a16, w8a16, w8a8 and w4a16, the quantizable data types of the set communication to-be-quantized objects include 16bit and 8bit, the data types of the KV cache to-be-quantized objects include 16bit, 8bit and 4bit, and the data types of the attention calculation to-be-quantized objects include 16bit and 8bit. As known from the embodiment of FIG. 4, there are four linear transformation to-be-quantized objects, each of which has four quantizable data types, there are two set communication to-be-quantized objects, each of which has two quantizable data types, there is one KV cache to-be-quantized object, which has three quantizable data types, and there is one attention calculation to-be-quantized object, which has two quantizable data types. In total, there are eight to-be-quantized objects, and free combination can obtain 6144 quantization strategies. The above examples are used for illustration, and the present application is not limited specifically.

[0125] It should be understood that each network layer includes 8 objects to be quantized, and if each object to be quantized in each network layer is freely combined according to different data types, the number of quantization strategies combined can be very large, and the calculation of the sensitivity of each quantization strategy can cause calculation pressure. Therefore, in the same way that the data types of objects to be quantized corresponding to the same type are combined into multiple quantization strategies, the calculation pressure can be effectively reduced.

[0126] In S340, the model quantization system 200 determines the sensitivity of each quantization strategy, and obtains a first quantization strategy based on the sensitivity.

[0127] In the embodiments of the present application, the model quantization system 200 can calculate the sensitivity of each quantization strategy, which is used to indicate the influence of the quantization strategy on the model accuracy. The model quantization system 200 can select a quantization strategy with the smallest influence on the model accuracy as the first quantization strategy from the multiple quantization strategies based on the sensitivity. In specific implementations, the model quantization system 200 can first calculate the sensitivity of each object to be quantized in the quantization strategy, and then determine the sensitivity of the entire quantization strategy based on the sensitivity of each object to be quantized, such as summing the sensitivity of each object to be quantized to obtain the sensitivity of the entire quantization strategy, or weighted sum to obtain the sensitivity of the entire quantization strategy, which is not limited in the present application. In specific implementations, if the sensitivity is greater, it means that the quantization strategy has a greater influence on the model accuracy. At this time, the sensitivities of multiple quantization strategies can be sorted, and the quantization strategy with the lowest sensitivity can be selected as the first quantization strategy.

[0128] Further, the sensitivity can be obtained based on one or more sensitive parameters related to the model accuracy. When calculating the sensitivity of each object to be quantized, one or more sensitive parameters can be calculated first, and then the sensitivity can be determined, such as obtaining the sensitivity according to the product of one or more sensitive parameters, or obtaining the sensitivity according to the weighted sum of one or more sensitive parameters, which is not limited in the present application.

[0129] Optionally, the sensitive parameters can include the average trace of the hessian matrix of the weights or activation values in the object to be quantized. It should be understood that if the average trace of the hessian matrix of the weights is smaller, the precision loss caused by the quantization of the weights to the entire model is greater. Therefore, the average trace as a sensitive parameter can well indicate the influence of the quantization strategy on the model accuracy. The activation value is similar to the weight, so the average trace of the activation value can also well indicate the influence of the quantization on the model accuracy. Similarly, the weights, activation values in the set communication object to be quantized, the attention calculation object to be quantized, and the KV cache object to be quantized can also be evaluated by the average trace to evaluate the influence of the quantization strategy on the model accuracy.

[0130] In a specific implementation, when calculating the average trace of the Hessian matrix of the weight of the linear transformation quantization operator, it is considered that the calculation of the average trace of the Hessian matrix is time-consuming, the trace(H) (indicating the trace of the Hessian matrix) of a large matrix is difficult to calculate, but the bilinear calculation form zTHz (z is a random vector) is relatively easy to calculate. Therefore, the Hutchinson algorithm can be used to quickly estimate the average trace trace(H) of the Hessian matrix of the weight based on the average value E(zTHz) of the form zTHz. Specifically, the formula can be as follows formula (1):

[0131] m random vectors zi conforming to the Gaussian distribution are selected, H is the Hessian matrix of the weight, and the average trace of the Hessian matrix can be quickly calculated by formula (1) based on the average trace of the weight Hessian matrix.

[0132] In a specific implementation, when calculating the average trace of the Hessian matrix of the activation value of the linear transformation quantization operator, since each input of the model is independent, the Hessian matrix of the intermediate layer activation is a block diagonal matrix, each block corresponds to the Hessian matrix of one input xi or another input xi+1, and therefore the average trace of the Hessian matrix of the activation layer can be obtained by averaging the traces of the Hessian matrices corresponding to the typical model inputs. Based on the Hutchinson algorithm, by selecting a certain number of inputs, the average value of the Hessian trace of the activation gradually converges, and therefore the average trace of the Hessian matrix of the activation value can be quickly and accurately estimated based on formula (2):

[0133] where N is the number of model inputs, x i is the i-th input of the model, z i is an element of a random vector z conforming to the Gaussian distribution, H aj(xi) is the Hessian matrix of the activation of the j-th layer of the model input x i .

[0134] The average trace of the Hessian matrix of the activation value and the weight of the linear transformation quantization operator is described in detail above. Since the input data of the KV cache to be quantized object, the attention calculation to be quantized object, and the set communication to be quantized object are also dynamically changed, they belong to the activation value. Referring to the related description of formula (2) above, the average trace of the Hessian matrix of the activation value of the KV cache to be quantized object, the attention calculation to be quantized object, and the set communication to be quantized object can be obtained. Specifically, for the KV cache to be quantized object, the sensitivity can be obtained based on the average trace of the Hessian matrix of the K vector and the V vector stored in the KV cache. For the attention calculation to be quantized object, the sensitivity can be obtained based on the average trace of the Hessian matrix of the Q vector, the K vector, and the V vector participating in the attention calculation. For the set communication to be quantized object, the sensitivity can be obtained based on the average trace of the Hessian matrix of the to-be-communicated data participating in the set communication. Here, examples are not given one by one.

[0135] It should be understood that the above exemplary gives the calculation method of the average trace of the Hessian matrix, and in the specific implementation, the average trace can also be calculated by other methods, which are not given one by one here.

[0136] Optionally, the sensitivity parameter can also include a loss value, that is, a loss function value calculated according to the output result before and after the model quantization, such as calculating the mean squared error (Mean Squared Error Loss, MSE) value under two precisions after converting the 32-bit floating point number to the 16-bit floating point number. The above examples are used for illustration, and the present application is not limited specifically.

[0137] It should be understood that the above exemplary gives some possible sensitivity parameters, and in the specific implementation, other sensitivity parameters can also be combined to determine the sensitivity, which is not limited specifically by the present application. After determining the sensitivity parameter of each to-be-quantized object, the corresponding sensitivity can be obtained by a certain calculation method, such as multiplying the average trace and the loss value to obtain the corresponding sensitivity, or obtaining the corresponding sensitivity by weighted summation, which is not limited specifically by the present application.

[0138] Optionally, after obtaining the sensitivity of each to-be-quantized object, the sensitivity of the entire quantization strategy can be determined according to the sensitivity of each to-be-quantized object. As known from the foregoing, the quantization strategy is freely combined, and if it is the large language model shown in FIG. 4, including 8 to-be-quantized objects, that is, 1 KV cache to-be-quantized object, 1 attention calculation to-be-quantized object, 2 set communication to-be-quantized objects, and 4 linear transformation to-be-quantized objects, and each to-be-quantized object can be quantized to the data type as described in the foregoing example, then there are 6144 kinds of quantization strategies in the free combination, and the sensitivity Ω corresponding to each quantization strategy can be calculated according to the following formula (3):

[0139] wherein i represents the i th layer of the large language model, L represents the total number of layers of the large language model, is the average trace of the Hessian matrix of the 8 operators as a whole, is the L2 norm of the quantization perturbation of the i th layer, that is, the L2 norm of the error between the output after quantization and the output before quantization, that is, the loss value in the foregoing. In simple terms, formula (3) multiplies the average trace of the Hessian matrix of the weight or activation value with the loss value, and performs a whole-network summation to obtain the sensitivity of the entire quantization strategy. Each quantization strategy is calculated in this way, and the quantization strategy with the lowest sensitivity is the optimal first quantization strategy.

[0140] It should be understood that Ω reflects the sensitivity of the entire 8 to-be-quantized objects of the model as a whole, which is an accuracy error evaluation of the entire quantization strategy, and takes into account the cumulative error of various data type combinations on the accuracy of the entire network. The first quantization strategy determined in this way takes into account the error accumulation of the quantization of the 8 to-be-quantized objects, and is the best optimization scheme to obtain the most suitable quantization data type for each to-be-quantized object.

[0141] S350: The model quantization system 200 evaluates the first quantization strategy to obtain a quantization evaluation result.

[0142] Optionally, the model quantization system 200 can first perform outlier suppression based on the first quantization strategy to obtain a suppressed first quantization strategy, and then perform quantization evaluation on the suppressed first quantization strategy to obtain a quantization evaluation result. Outlier suppression refers to identifying and adjusting extreme outliers in the data to reduce the impact of these outliers on the quantization process. Outliers refer to values that are significantly different from the overall distribution of the data. Outliers may be caused by some small probability events and do not have representativeness, but will have an impact on the overall data distribution.

[0143] In specific implementation, the model quantization system 200 can obtain the data distribution of the activation value of the to-be-quantized object based on model forward inference, identify outliers in the data distribution based on an outlier identification algorithm, and then adjust the outliers based on an outlier suppression algorithm to make the data distribution more gentle, thereby obtaining a first quantization strategy after suppression. The outlier identification algorithm can include an interquartile range method (IQR), a standard score method (Z-score) algorithm, etc., and detect discrete values deviating from the main distribution, which can adversely affect the accuracy of quantization. The outlier suppression algorithm can use a mathematical equivalent change method, such as stage or tail reduction processing on extreme activation values, to make the data distribution more gentle and reduce quantization errors.

[0144] Optionally, when the first quantization strategy is quantization evaluated, quantization calibration can be performed first to obtain a quantization parameter of each to-be-quantized object, and then the to-be-quantized object is quantized using the quantization parameter to obtain a quantized to-be-evaluated model, and the to-be-evaluated model is run on a test set to obtain a quantization evaluation result. The quantization evaluation result can include accuracy, inference speed, resource consumption, model file size, robustness, stability, etc., which are not limited in the present application.

[0145] In specific implementation, quantization calibration is used to determine quantization parameters to reduce accuracy loss in the quantization process. The quantization parameters can include a scale and an offset. The scale is used to adjust the proportion of data in the quantization process, so that the data can be mapped from a high-precision format to a low-precision format range. For example, if the first range of the original data is from -1.5 to 1.5, and the second range of the quantized data is from 0 to 255, the scale is used to map the first range to the second range. The offset is used to adjust the center point of the data to a suitable position to ensure that the quantized data can reflect the statistical characteristics of the original data as much as possible. The algorithm used to determine the quantization parameters can include a Min-Max Scaling method, a KL Divergence, etc., and can also include other methods for effectively mapping data to a low-precision numerical orientation, which are not limited in the present application.

[0146] It should be understood that although the quantized model uses a low-precision quantization data type, the model output can not be greatly affected by determining appropriate quantization parameters through quantization calibration. In addition, the quantization calibration of multiple to-be-quantized objects can be performed simultaneously, sequentially, partially simultaneously and partially sequentially, which can be set according to actual business requirements, which are not limited in the present application.

[0147] S360: The model quantization system 200 determines whether the quantization evaluation result is qualified. In the case of yes, that is, in the case of a qualified quantization evaluation result, S380 is performed, and in the case of no, that is, in the case of an unqualified quantization evaluation result, S370 is performed.

[0148] Optionally, based on the quantization evaluation result, the pre-quantization indicators and the post-quantization indicators can be compared to obtain indicator loss values, and based on the gap between the indicator loss values and the threshold values, it is determined whether the quantization evaluation result is qualified. For example, the pre-quantization model precision (such as classification precision, regression precision, etc.) and the post-quantization model precision are compared to obtain a precision loss value, the pre-quantization model stability score (such as noise resistance ability, generalization ability) and the post-quantization model stability score are compared to obtain a stability loss value, etc. Based on the loss values of multiple indicators, it is determined whether the quantization evaluation result is qualified. Further, the quantization evaluation result can also be scored based on a pre-set evaluation system, and based on the score, it is determined whether the quantization evaluation result is qualified. The present application is not limited in detail.

[0149] S370: The model quantization system 200 performs quantization rollback on the first quantization strategy to obtain a second quantization strategy.

[0150] In specific implementations, if the quantization evaluation result shows that the first quantization strategy is unqualified, the quantization rollback can be performed on the first quantization strategy to rollback the quantization data types of part of the to-be-quantized objects in the first quantization strategy to higher precision to obtain a second quantization strategy with a qualified evaluation result. As known from the foregoing, in the first quantization strategy, the quantization data types of the same to-be-quantized objects in each network layer are the same, and therefore S370 can perform quantization rollback according to the network layers, sequentially determine the quantization data types of each to-be-quantized object in each network layer, and until a second quantization strategy with a qualified evaluation result is obtained. In this way, the second quantization strategy finally obtained may have different quantization data types for the same to-be-quantized object in different network layers, such as 16-bit quantization data types for the KV cache to-be-quantized object in the first network layer and 8-bit quantization data types for the KV cache to-be-quantized object in the second network layer. The above examples are used for illustration, and the present application is not limited in detail.

[0151] It should be understood that the present application first screens out a relatively excellent first quantization strategy through steps S310-S360, and in the first quantization strategy, the quantization data types of the same to-be-quantized object in each network layer are the same. Then, through S370, quantization rollback is performed according to each network layer to obtain a second quantization strategy with a finer granularity. Compared with the traditional way of sequentially searching for corresponding quantization data types for a large number of parameters, the way of coarse-grained screening and then fine-grained rollback can reduce the search space and thus improve the efficiency of large language model quantization.

[0152] Optionally, in order to further reduce the search space, the model quantization system 200 can pre-set M fallback schemes, each of which includes a to-be-quantized object participating in quantization and a data type to which the to-be-quantized object can be quantized. The to-be-quantized objects included in different fallback schemes can be different, and the data types to which the to-be-quantized objects can be quantized can also be different. For example, fallback scheme 1 includes to-be-quantized object 1 and to-be-quantized object 2, fallback scheme 2 includes to-be-quantized object 1, to-be-quantized object 2, and to-be-quantized object 3, fallback scheme 3 includes to-be-quantized object 1 and to-be-quantized object 3, and so on. Then, according to a search order, the optimal quantization strategy under each fallback scheme is determined in sequence, and the optimal quantization strategy is quantization evaluated. If the evaluation result is qualified, the quantization strategy is determined as the second quantization strategy. If it is not qualified, the optimal quantization strategy under the next fallback scheme is determined, and so on, until the second quantization strategy with a qualified evaluation result is obtained. In this way, the second quantization strategy can be obtained by evaluating at most M times, which can improve the efficiency of quantization fallback.

[0153] In a specific implementation, when the optimal quantization strategy under each fallback scheme is determined, the sensitivity of the to-be-quantized object can also be calculated according to the manner of determining the first quantization strategy described above, and the optimal quantization strategy under the fallback scheme is obtained based on the sensitivity sorting. It should be noted that when the optimal quantization strategy under each fallback scheme is determined, the quantization strategy of each network layer is also determined based on the sensitivity sorting manner, and then the optimal quantization strategy under the fallback scheme is obtained.

[0154] For example, it is assumed that the optimal quantization strategy in fallback scheme 1 is determined. Fallback scheme 1 includes a KV cache to-be-quantized object, a linear transformation to-be-quantized object, and an attention calculation to-be-quantized object. At this time, the number of to-be-quantized objects is 6, and the data types of the 8 to-be-quantized objects in the first network layer can be freely combined to obtain m1 quantization strategies. Then, the sensitivity of the m1 quantization strategies in this network layer is calculated according to the above formulas (1)-(3), and then the optimal quantization strategy is determined. After the optimal quantization strategies of the N network layers are obtained, the optimal quantization strategy of fallback scheme 1 is obtained, which includes the quantization data type of each to-be-quantized object in each network layer. The quantization data type of the same to-be-quantized object in different network layers can be different. At this time, the optimal quantization strategy is quantization evaluated. If it is qualified, it can be determined as the second quantization strategy. If it is not qualified, the optimal quantization strategy in fallback scheme 2 is determined. The determination process can be referred to the description above, which will not be described here.

[0155] Further, the search order of the fallback scheme can be determined according to the number of objects to be quantized in the fallback scheme. The more the number of objects to be quantized in the fallback scheme, the higher the search order, in other words, when searching, gradually reduce the number of objects to be quantized participating in the search, and gradually increase the number of objects to be quantized in the fallback. For example, there are 5 objects to be quantized, first search according to 4 objects to be quantized, that is, fallback 1 object to be quantized, then search according to 3 objects to be quantized, that is, fallback 2 objects to be quantized, then search according to 2 objects to be quantized, that is, fallback 3 objects to be quantized, and so on. In this way, gradually reducing the number of objects to be quantized participating in the quantization can avoid large-scale fallback and improve the efficiency of fallback.

[0156] For example, in the scenario of FIG. 4, it is assumed that the first quantization strategy includes 8 objects to be quantized, and Table 1 is an example table of the fallback scheme. As shown in Table 1, scheme 5 and scheme 4 include 3 objects to be quantized, scheme 3, scheme 2 and scheme 1 include 2 objects to be quantized. Therefore, when quantization fallback is performed, the optimal quantization strategy (also referred to as the third quantization strategy) under each fallback scheme can be determined in the order of scheme 5, scheme 4, scheme 3, scheme 2 and scheme 1, and then the quantization strategy is evaluated. If it is not qualified, the optimal quantization strategy under the next fallback scheme is determined, until a second quantization strategy with a qualified evaluation result is obtained. In this way, the second quantization strategy can be obtained by evaluating at most 5 times, reducing the number of quantization evaluations, and improving the quantization efficiency.

[0157] Table 1: Example table of fallback scheme

[0158] It should be noted that if no quantization strategy with a qualified evaluation result is obtained under the last fallback scheme, a notification message of manual quantization fallback can be sent to the user to prompt the user to perform manual fallback. The user can manually set the second quantization strategy according to experience.

[0159] In a specific implementation, when prompting the user to perform manual fallback, the prompt message can include the optimal quantization strategy under each fallback scheme and the quantization evaluation result. Although the evaluation results of these quantization strategies are not qualified, they can be used as a reference for the user to obtain the quantization strategy of manual fallback more quickly. Further, the quantization evaluation results of the optimal quantization strategies under each fallback scheme can be analyzed, and then the analysis results are displayed to the user, such as the mean square error of each quantization strategy, so that the user can determine which objects to be quantized and which network layers have larger errors, and modify the quantization data type of the object to be quantized to obtain the second quantization strategy.

[0160] It should be understood that, when quantization backtracking, the quantization strategy is searched in a more fine-grained manner, and the fine-grained search is completed in a certain search order, which can not only obtain a more fine and accurate quantization strategy, but also improve the search efficiency. Compared with the scheme of directly searching in the massive search space of the entire multi-network layer, multiple quantization objects, and multiple data types, the technical scheme of the present application can more efficiently and accurately obtain the optimal quantization strategy, thereby improving the quantization efficiency of the model.

[0161] S380: The model quantization system 200 quantizes the model to be quantized to obtain a model quantization result.

[0162] In a specific implementation, if the first quantization strategy evaluation result is qualified, the first quantization strategy can be used to quantize the model to be quantized to obtain a model quantization result, and S360 need not be executed. If the first quantization strategy evaluation result is not qualified, the second quantization strategy determined in S360 can be used for quantization to obtain a model quantization result.

[0163] In a specific implementation, the model quantization result includes quantized model data, quantization strategy, quantization weight, quantization parameter, and the like. The specific implementation can be determined according to actual business requirements, and the present application is not limited in this regard.

[0164] S390A: The model quantization system 200 sends a quantization report to the client 100.

[0165] In a specific implementation, the quantization report can be generated based on the model quantization result. The quantization report can include the model quantization result, information related to the model before quantization, and analysis data for analyzing the model quantization result. For example, some index data in the quantization evaluation result is processed in a chart manner to analyze some possible risks of the quantized model and propose corresponding risk control measures. The specific content included in the quantization report can be determined according to actual business requirements, and the present application is not limited in this regard.

[0166] S390B: The model quantization system 200 sends the model quantization result to the inference system 300.

[0167] Optionally, the model quantization system 200 can simultaneously execute S390A and S390B, or after S390A is executed, the user determines to use the second quantization strategy, and after sending a confirmation message to the model quantization system 200 through the client 100, the model quantization system 200 executes S390B. The present application is not limited in this regard.

[0168] It should be understood that after the inference system 300 receives the model quantization result, the inference engine can load the quantized weights and parameters according to the quantized weights and quantized parameters therein, and perform inference deployment. The quantized model can reduce the required memory space of the model, so that the model is more easily deployed in memory-constrained devices, such as mobile devices, embedded systems, edge devices, etc., and can accelerate the inference process, which is particularly important for applications that require real-time processing, such as real-time translation, dialogue systems, etc., and can also reduce the energy consumption of the device required for model running, which is a good advantage for battery-powered mobile devices.

[0169] In summary, the present application provides a method for determining a quantization strategy. The method obtains a plurality of different quantization strategies by freely combining the to-be-quantized objects supported by the to-be-quantized model and the quantizable data types, and then determines the sensitivity of different quantization strategies based on a sensitivity algorithm. The sensitivity is used to indicate the influence of the quantization strategy on the accuracy of the to-be-quantized model. The first quantization strategy is determined according to the sensitivity of each quantization strategy. Then, the first quantization strategy is quantized and evaluated. If the evaluation is qualified, the first quantization strategy is determined as the quantization strategy of the to-be-quantized model, so that the entire quantization process does not need to be quantized and searched, thereby solving the problem of low quantization efficiency caused by long time consumption of quantization search in model quantization. If the evaluation is not qualified, the first quantization strategy is quantized and rolled back based on the first quantization strategy, the accuracy of part of the to-be-quantized objects in the first quantization strategy is improved, until a second quantization strategy that passes the evaluation is obtained. Compared with the traditional quantization search method based on a large number of parameters, the method of determining the first quantization strategy and then rolling back can reduce the search space, thereby solving the problem of large search space and long search time when searching for a quantization strategy in the model quantization process, which leads to low model quantization efficiency.

[0170] At the same time, evaluating the accuracy in units of quantization strategies can avoid the problem of model accuracy exceeding the controllable range due to the accumulation of accuracy caused by quantizing a plurality of to-be-quantized objects, so that the entire model can quantize a plurality of to-be-quantized objects at multiple calculation stages, and the quantization degree of the model can be maximized to develop the potential of the overall model quantization.

[0171] The above describes in detail the architecture of the model quantization system and the method for determining a quantization strategy provided by the present application. The internal structure of the model quantization system provided by the present application is explained and described below in conjunction with FIG. 5.

[0172] FIG. 5 is a structural schematic diagram of a model quantification system provided by the present application, which can be the model quantification system in the foregoing. As shown in FIG. 5, the model quantification system 200 includes an obtaining unit 510, a combining unit 520, a determining unit 530, an evaluation unit 540, a fallback unit 550, a quantification unit 560, and a prompting unit 570. It should be understood that FIG. 5 is an exemplary division manner, and the model quantification system 200 can be divided into more or fewer unit modules, which is not specifically limited by the present application.

[0173] The obtaining unit 510, the combining unit 520, the determining unit 530, the evaluation unit 540, the fallback unit 550, the quantification unit 560, and the prompting unit 570 can be implemented by software or by hardware. For example, the implementation of the obtaining unit 510 is described below. Similarly, the implementation of the combining unit 520, the determining unit 530, the evaluation unit 540, the fallback unit 550, the quantification unit 560, and the prompting unit 570 can refer to the implementation of the obtaining unit 510.

[0174] As an example of the software function unit, the obtaining unit 510 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the obtaining unit 510 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code can be distributed in the same region, or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers for running the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0175] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same virtual private cloud (VPC), or can be distributed in multiple VPCs. Generally, one VPC is set in one region, and a communication gateway needs to be set in each VPC for cross-region communication between two VPCs in the same region and between VPCs in different regions, so as to realize the interconnection between the VPCs through the communication gateway.

[0176] As an example of the hardware function unit, the obtaining unit 510 can include at least one computing device, such as a server, etc. Alternatively, the obtaining unit 510 can also be a device implemented by a central processing module (CPU), or implemented by an application-specific integrated circuit (ASIC), or implemented by a programmable logic device (PLD), etc. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing module (DPU), a neural network processing module (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0177] The multiple computing devices included in the obtaining unit 510 can be distributed in the same region, or distributed in different regions. The multiple computing devices included in the obtaining unit 510 can be distributed in the same AZ, or distributed in different AZs. Similarly, the multiple computing devices included in the obtaining unit 510 can be distributed in the same VPC, or distributed in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, etc.

[0178] It should be noted that in other embodiments, the steps implemented by the obtaining unit 510, the combining unit 520, the determining unit 530, the evaluation unit 540, the fallback unit 550, the quantization unit 560, and the prompting unit 570 can be specified as needed, and the overall function of the model quantization system 200 can be implemented by the different steps in the method for determining the quantization strategy implemented by the obtaining unit 510, the combining unit 520, the determining unit 530, the evaluation unit 540, the fallback unit 550, the quantization unit 560, and the prompting unit 570, respectively.

[0179] The functions of the obtaining unit 510, the combining unit 520, the determining unit 530, the evaluation unit 540, the fallback unit 550, the quantization unit 560, and the prompting unit 570 will be explained below.

[0180] The acquisition unit 510 is used to acquire one or more objects to be quantized from the model to be quantized, and at least one data type that each object to be quantized can be quantized. Specifically, it is used to implement S310, S320 and their optional steps in the embodiment of Figure 3.

[0181] The combination unit 520 is used to determine multiple quantization strategies based on at least one data type that each object to be quantized can be quantized, wherein each quantization strategy includes a quantization data type for each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can be quantized. Specifically, it is used to implement S330 and its optional steps in the embodiment of FIG3.

[0182] The determining unit 530 is used to determine the sensitivity of each quantization strategy, wherein the sensitivity is used to indicate the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy. Based on the sensitivity of each quantization strategy, a first quantization strategy is obtained from multiple quantization strategies, wherein the first quantization strategy is used to quantize one or more objects to be quantized, specifically to implement S340 and its optional steps in the embodiment of Figure 3.

[0183] In one possible implementation, the determining unit 530 is used to determine the sensitivity of each object to be quantized in each quantization strategy, and to obtain the sensitivity of each quantization strategy based on the sensitivity of each object to be quantized. The sensitivity of each object to be quantized is obtained based on one or more parameters, including the average trace of the Hessian matrix of the object to be quantized and the loss values ​​of the object before and after quantization. Specifically, this is used to implement S340 and its optional steps in the embodiment of Figure 3.

[0184] In one possible implementation, the evaluation unit 540 is used to perform a quantitative evaluation on the first quantization strategy and obtain a quantitative evaluation result. Specifically, it is used to implement steps S350-S360 and their optional steps in the embodiment of Figure 3. The rollback unit 550 is used to perform a quantitative rollback on the first quantization strategy when the quantitative evaluation result is unqualified. It modifies the quantization data type of the object to be quantized in the first quantization strategy to obtain a second quantization strategy. The second quantization strategy is used to quantize one or more objects to be quantized. Specifically, it is used to implement step S380 and its optional steps in the embodiment of Figure 3.

[0185] In one possible implementation, the quantization unit 560 is used to quantize the model to be quantized using a first quantization strategy or a second quantization strategy to obtain a quantized model, and then send the quantized model to the inference system. Specifically, it is used to implement steps S380, S390A, S390B and their optional steps in the embodiment of Figure 3.

[0186] In a possible implementation, the to-be-quantized model includes a plurality of network layers, in the first quantization strategy, a same to-be-quantized object in each network layer has a same quantization data type, and in the second quantization strategy, the same to-be-quantized object in each network layer has a same or different quantization data type, and a quantity of to-be-quantized objects in the second quantization strategy is less than a quantity of to-be-quantized objects in the first quantization strategy.

[0187] In a possible implementation, the fallback unit 550 is configured to obtain a plurality of fallback schemes, each of which includes at least one to-be-quantized object, and different fallback schemes include different quantities of to-be-quantized objects, determine a quantization data type of each to-be-quantized object in a first fallback scheme in the plurality of fallback schemes, obtain a third quantization strategy corresponding to the first fallback scheme, perform quantization evaluation on the third quantization strategy to obtain a fallback evaluation result, determine the third quantization strategy as a second quantization strategy in a case where the fallback evaluation result is qualified, determine a quantization data type of each to-be-quantized object in a second fallback scheme in a case where the fallback evaluation result is unqualified, and obtain a fourth quantization strategy corresponding to the second fallback scheme, where a quantity of to-be-quantized objects in the first fallback scheme is not less than a quantity of to-be-quantized objects in the second fallback scheme.

[0188] In a possible implementation, the fallback unit 550 is configured to obtain, based on to-be-quantized objects included in the first fallback scheme and a data type that can be quantized for each to-be-quantized object, at least one hierarchical quantization strategy of each network layer under the first fallback scheme, determine a sensitivity of each hierarchical quantization strategy of each network layer, determine a quantization strategy of each network layer based on the sensitivity of each hierarchical quantization strategy, and obtain a third quantization strategy corresponding to the first fallback scheme based on the quantization strategy of each network layer.

[0189] In a possible implementation, the to-be-quantized object includes a linear transformation to-be-quantized object, a key-value cache (KV cache) to-be-quantized object, a set communication to-be-quantized object, and an attention calculation to-be-quantized object, the linear transformation to-be-quantized object includes an operator performing linear transformation on data, an activation value parameter, and a weight parameter, the KV cache to-be-quantized object includes a K vector and a V vector in the KV cache, the attention calculation to-be-quantized object includes an operator performing matrix multiplication calculation in an attention layer, and the set communication to-be-quantized object includes communication data in set communication.

[0190] In a possible implementation, the prompting unit 570 is configured to analyze the rollback evaluation result corresponding to each rollback scheme in a case where the rollback evaluation result of each rollback scheme is all unqualified, and obtain an analysis result, the analysis result including the to-be-quantized object and the network layer information whose influence on the quantization error is greater than a threshold, and the prompting unit 570 is configured to send a prompt message of manual rollback to the client, the prompt message including one or more of the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and the analysis result.

[0191] In conclusion, the model quantization system provided in this application can obtain a plurality of different quantization strategies by freely combining the to-be-quantized objects supported by the to-be-quantized model and the quantifiable data types, and determine the sensitivity of different quantization strategies based on a sensitivity algorithm, the sensitivity being used to indicate the influence of the quantization strategy on the precision of the to-be-quantized model, determine a first quantization strategy according to the sensitivity of each quantization strategy, and then perform quantization evaluation on the first quantization strategy, if the evaluation is qualified, determine the first quantization strategy as the quantization strategy of the to-be-quantized model, so that the entire quantization process does not need to perform quantization search, thereby solving the problem of low quantization efficiency caused by long time consumption of quantization search in model quantization; if the evaluation is unqualified, perform quantization rollback based on the first quantization strategy, improve the precision of part of the to-be-quantized objects in the first quantization strategy, until a second quantization strategy that passes the evaluation is obtained. Compared with the traditional quantization search method based on large-scale parameters, the method of first determining the first quantization strategy and then rollback can reduce the search space, thereby solving the problem of large search space and long search time when searching for a quantization strategy in the model quantization process, and the problem of low model quantization efficiency.

[0192] Meanwhile, evaluating the precision in units of quantization strategies can avoid the problem of model precision exceeding the controllable range caused by the accumulation of the precision of a plurality of to-be-quantized objects, so that the entire model can quantize the to-be-quantized objects in multiple calculation stages at the same time, and the quantization degree of the model can be maximized, thereby developing the potential of the overall model quantization.

[0193] The method for determining a quantization strategy and the model quantization system provided in this application are described in detail above, and the computing device provided in this application is explained and described below in combination with FIGS. 6-8.

[0194] FIG. 6 is a structural schematic diagram of a computing device provided by the present application. As shown in FIG. 6, the computing device 600 includes a bus 602, a processor 604, a memory 606, an acceleration card 607, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate through the bus 602. The computing device 600 can be a server. It should be understood that the number of processors, memories, and acceleration cards in the computing device 600 is not limited by the present application. The computing device 600 can be used to deploy the model quantization system in the foregoing embodiments of FIGS. 1-5. The memory 606 is configured to store instructions, and the processor 604 is configured to execute the instructions stored in the memory 606. The memory 606 stores program codes, and the processor 604 can invoke the program codes stored in the memory 606 to perform the following operations:

[0195] obtain one or more to-be-quantized objects of a to-be-quantized model, and at least one data type that each to-be-quantized object can be quantized to;

[0196] determine a plurality of quantization strategies based on the at least one data type that each to-be-quantized object can be quantized to, wherein each quantization strategy includes a quantization data type of each to-be-quantized object, and the quantization data type is one of the at least one data type that the to-be-quantized object can be quantized to;

[0197] determine a sensitivity of each quantization strategy, wherein the sensitivity is used to indicate a degree of accuracy loss of the to-be-quantized model after quantization using the quantization strategy;

[0198] obtain a first quantization strategy from the plurality of quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize the one or more to-be-quantized objects.

[0199] It should be understood that the memory 606 stores executable program codes, and the processor 604 executes the executable program codes to implement the operation steps of the method for determining a quantization strategy described in each of the embodiments of FIGS. 1 and 5. The program codes can include one or more software units, and the one or more software units are the obtaining unit, the combining unit, the determining unit, the evaluating unit, the fallback unit, the quantizing unit, and the prompting unit in the embodiment of FIG. 5, wherein the obtaining unit is used to implement S310 and S320 and the optional steps thereof in the embodiment of FIG. 3, the combining unit is used to implement S330 and the optional steps thereof in the embodiment of FIG. 3, the determining unit is used to implement S340 and the optional steps thereof in the embodiment of FIG. 3, the evaluating unit is used to implement S350 and the optional steps thereof in the embodiment of FIG. 3, the fallback unit is used to implement S360 and the optional steps thereof in the embodiment of FIG. 3, the quantizing unit is used to implement S370 and the optional steps thereof in the embodiment of FIG. 3, and the prompting unit is used to implement S390A-S390B and the optional steps thereof in the embodiment of FIG. 3. Details of each unit are not described herein.

[0200] The bus 602 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (UBus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The unified bus can also be referred to as a coherent accelerated processor bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is represented in FIG. 6, but it does not mean that there is only one bus or only one type of bus. The bus 602 can include a path for transmitting information between various components (e.g., the memory 606, the processor 604, the communication interface 608) of the computing device 600. The unified bus can also be referred to as a coherent accelerated processor bus.

[0201] The processor 604 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, a FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, or the like computing device.

[0202] The memory 606 can include a volatile memory, such as a random access memory (RAM) including a dynamic or static RAM. The memory 606 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). In addition, the memory 606 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0203] It is worth mentioning that the same type of storage medium can be configured in the same computing device to implement the function of the memory 606, or two or more types of storage media can be configured to implement the function of the memory 606, which is not limited in the present application.

[0204] The communication interface 608 uses a transceiver module such as but not limited to a network interface card, a transceiver, etc. to realize the communication between the computing device 600 and other devices or communication networks.

[0205] As a possible implementation manner, the method shown in FIGS. 1-5 can be implemented by the acceleration card 607 in the computing device 600, in addition to being implemented by the processor in the computing device 600 executing the program code in the memory, wherein the acceleration card 607 can be implemented by a computing device such as a DPU, NPU, GPU, XPU, SoC, etc., or an AI chip, etc., which is not limited in the present application. The acceleration card 607 includes a processing unit and a power supply circuit, and the power supply circuit is used to power the processing unit, so that the processing unit executes the operation steps of the method for determining the quantization strategy described in each embodiment of FIGS. 1-5.

[0206] As a possible implementation manner, in addition to being implemented by the processor in the computing device 600 executing the program code in the memory, or by the acceleration card 607, the method described in FIGS. 1-5 can also be implemented by the combination of the processor 604 and the acceleration card 607 in the computing device 600 to execute the operation steps performed by the corresponding subject in the method for determining the quantization strategy described in FIGS. 1-5. Specifically, the program code stored in the memory 606 includes part of the unit modules shown in FIG. 5, so that the processor 604 executes the program code to implement part of the operation steps of the method for determining the quantization strategy, and the power supply circuit in the acceleration card 607 powers the processing unit, so that the processing unit executes another part of the operation steps.

[0207] For example, the memory 606 stores program codes of the obtaining unit, the combining unit, and the prompting unit, and the processing unit of the acceleration card 607 is used to implement the functions of the determining unit, the evaluating unit, the quantizing unit, and the falling back unit. The above examples are used for illustration, and the application is not limited thereto, and the function descriptions of the above-mentioned units can refer to the related content described above, which will not be repeated and described here.

[0208] The embodiments of the present application also provide a computing device cluster, as shown in FIG. 7, which is an example diagram of a computing device cluster provided by the present application, which can include at least one computing device of the embodiment of FIG. 6, and the computing device cluster is used to deploy the model quantization system 200 described above.

[0209] As a possible implementation, the memory 606 in one or more of the computing devices 600 in the computing device cluster can have instructions for implementing the same operational steps of the method for determining the quantization strategy described in FIG. 1 to FIG. 5. Alternatively, the memory 606 in one or more of the computing devices 600 can have instructions for implementing different operational steps of the method for determining the quantization strategy described in FIG. 1 to FIG. 5, respectively, in other words, a combination of one or more of the computing devices 600 can collectively implement the instructions for implementing the operational steps of the method for determining the quantization strategy.

[0210] As a possible implementation, the accelerator card 607 in one or more of the computing devices 600 in the computing device cluster can have instructions for implementing the same operational steps of the method for determining the quantization strategy described in FIG. 1 to FIG. 5. Alternatively, the accelerator card 607 in one or more of the computing devices 600 can have instructions for implementing different operational steps of the method for determining the quantization strategy described in FIG. 1 to FIG. 5, respectively, in other words, a combination of one or more of the computing devices 600 can collectively implement the instructions for implementing the operational steps of the method for determining the quantization strategy.

[0211] As a possible implementation, the memory 606 in one or more of the computing devices 600 in the computing device cluster can have instructions for implementing some of the operational steps of the method for determining the quantization strategy described in FIG. 1 to FIG. 5, and the accelerator card 607 can have instructions for implementing the other operational steps. Alternatively, the memory 606 in one or more of the computing devices 600 can have instructions for implementing different operational steps of the method for determining the quantization strategy described in FIG. 1 to FIG. 5, respectively, and the accelerator card 607 can have instructions for implementing different operational steps, in other words, a combination of one or more of the computing devices 600 can collectively implement the instructions for implementing the operational steps of the method for determining the quantization strategy.

[0212] In some possible implementations, one or more of the computing devices in the computing device cluster can be connected through a network. Here, the network can be a wide area network, a local area network, or the like. FIG. 8 shows a possible implementation. As shown in FIG. 8, FIG. 8 is a structural diagram of another computing device cluster provided by the present application. Two computing devices 600A and 600B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this type of possible implementation, the memory 606 in the computing device 600B has instructions for implementing the functions of the obtaining unit, the combining unit, and the prompting unit. Meanwhile, the accelerator card 607 in the computing device 600A is used to implement the functions of the determining unit, the evaluating unit, the falling back unit, and the quantizing unit.

[0213] The connection between the computing device clusters shown in FIG. 8 can be that, considering that the data query system is usually a distributed system, in order to reduce the processing pressure of each device, therefore, it is considered to distribute the unit modules on different computing devices.

[0214] It should be understood that the functions of the computing device 600A shown in FIG. 8 can also be completed by multiple computing devices 600. Similarly, the functions of the computing device 600B can also be completed by multiple computing devices 600.

[0215] The embodiments of the present application also provide a chip, which comprises a processing unit and a power supply circuit, and the power supply circuit is used to supply power to the processing unit, so that the processing unit performs the operation steps of the method for determining the quantization strategy in the embodiments of FIG. 1 to FIG. 5. For the sake of brevity, it will not be repeated here.

[0216] The embodiments of the present application also provide a chip system, which comprises a plurality of chips described above, and the chip system is used to deploy the model quantization system 200 in the foregoing.

[0217] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it makes the at least one computing device execute the method for determining the quantization strategy provided by the present application.

[0218] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like. The computer readable storage medium includes instructions, which instruct the computing device to execute the inference method based on the large language model, or instruct the computing device to execute the inference method.

[0219] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of determining a quantization policy, the method comprising: The method is applied to a model quantization system, and the method comprises: obtaining one or more to-be-quantized objects of a to-be-quantized model, and at least one data type that each to-be-quantized object can be quantized; determining a plurality of quantization strategies based on the at least one data type that each to-be-quantized object can be quantized, wherein each quantization strategy comprises a quantization data type of each to-be-quantized object, and the quantization data type is one of the at least one data type that the to-be-quantized object can be quantized; determining a sensitivity of each quantization strategy, wherein the sensitivity is used to indicate a degree of accuracy loss of the to-be-quantized model after quantization using the quantization strategy; obtaining a first quantization strategy from the plurality of quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize the one or more to-be-quantized objects.

2. The method of claim 1, wherein, The determination of the sensitivity of each quantization strategy comprises: determining a sensitivity of each to-be-quantized object in each quantization strategy, and obtaining the sensitivity of each quantization strategy based on the sensitivity of each to-be-quantized object, wherein the sensitivity of each to-be-quantized object is obtained based on one or more parameters of an average trace of a Hessian matrix of the to-be-quantized object and a loss value before and after quantization of the to-be-quantized object.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: performing quantization evaluation on the first quantization strategy to obtain a quantization evaluation result; in a case where the quantization evaluation result is unqualified, performing quantization rollback on the first quantization strategy, modifying the quantization data type of the to-be-quantized object in the first quantization strategy to obtain a second quantization strategy, and the second quantization strategy is used to quantize the one or more to-be-quantized objects.

4. The method of claim 3, wherein, The method further comprises: quantizing the to-be-quantized model using the first quantization strategy or the second quantization strategy to obtain a quantized model; sending the quantized model to an inference system.

5. The method according to claim 3 or 4, characterized in that, The to-be-quantized model comprises a plurality of network layers, the quantization data type of a same to-be-quantized object in each network layer is the same in the first quantization strategy, and the quantization data type of the same to-be-quantized object in each network layer is the same or different in the second quantization strategy, and the number of to-be-quantized objects in the second quantization strategy is less than the number of to-be-quantized objects in the first quantization strategy.

6. The method of claim 5, wherein, The quantization rollback on the first quantization strategy to obtain the second quantization strategy comprises: obtaining a plurality of rollback schemes, each rollback scheme comprising at least one to-be-quantized object, and the number of to-be-quantized objects included in different rollback schemes is different; determining a quantization data type of each to-be-quantized object in a first rollback scheme in the plurality of rollback schemes to obtain a third quantization strategy corresponding to the first rollback scheme; performing quantization evaluation on the third quantization strategy to obtain a rollback evaluation result; in a case where the rollback evaluation result is qualified, determining the third quantization strategy as the second quantization strategy. In a case where the rollback evaluation result is unqualified, a quantization data type of each to-be-quantized object in a second rollback scheme is determined, and a fourth quantization strategy corresponding to the second rollback scheme is obtained, where a number of to-be-quantized objects in the first rollback scheme is not less than a number of to-be-quantized objects in the second rollback scheme.

7. The method of claim 6, wherein, The determining of the quantization data type of each to-be-quantized object in the first rollback scheme from the plurality of rollback schemes and the obtaining of the third quantization strategy corresponding to the first rollback scheme include: obtaining, based on the to-be-quantized objects included in the first rollback scheme and at least one data type that can be quantized for each to-be-quantized object, at least one hierarchical quantization strategy of each network layer under the first rollback scheme; determining a sensitivity of each hierarchical quantization strategy of each network layer, and determining a quantization strategy of each network layer based on the sensitivity of each hierarchical quantization strategy; obtaining the third quantization strategy corresponding to the first rollback scheme based on the quantization strategy of each network layer.

8. The method according to any one of claims 1 to 7, characterized in that, The to-be-quantized objects include a linear transformation to-be-quantized object, a key-value cache (KV cache) to-be-quantized object, a collective communication to-be-quantized object, and an attention calculation to-be-quantized object, where the linear transformation to-be-quantized object includes an operator performing linear transformation on data, an activation value parameter, and a weight parameter, the KV cache to-be-quantized object includes a K vector and a V vector in the KV cache, the attention calculation to-be-quantized object includes an operator of matrix multiplication calculation in an attention layer, and the collective communication to-be-quantized object includes communication data in collective communication.

9. The method according to any one of claims 6 to 8, characterized in that, The method further includes: in a case where the rollback evaluation result of each rollback scheme is unqualified, analyzing the rollback evaluation result of each rollback scheme to obtain an analysis result, where the analysis result includes to-be-quantized object and network layer information whose influence on quantization error is greater than a threshold value; sending, to a client held by a user, a prompt message of manual rollback, where the prompt message includes the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and one or more of the analysis result.

10. A model quantization system, comprising: The system includes: an acquisition unit configured to acquire one or more to-be-quantized objects of a to-be-quantized model and at least one data type that can be quantized for each to-be-quantized object; a combination unit configured to determine a plurality of quantization strategies based on the at least one data type that can be quantized for each to-be-quantized object, where each quantization strategy includes a quantization data type of each to-be-quantized object, and the quantization data type is one of the at least one data type that can be quantized for the to-be-quantized object; a determination unit configured to determine a sensitivity of each quantization strategy, where the sensitivity is used to indicate a degree of accuracy loss of the to-be-quantized model after quantization using the quantization strategy; and a determination unit configured to acquire a first quantization strategy from the plurality of quantization strategies based on the sensitivity of each quantization strategy, where the first quantization strategy is used to quantize the one or more to-be-quantized objects.

11. The system of claim 10, wherein, The determining unit is configured to determine a sensitivity of each to-be-quantized object in each quantization strategy, and obtain the sensitivity of the quantization strategy based on the sensitivity of each to-be-quantized object, wherein the sensitivity of each to-be-quantized object is obtained based on one or more parameters of an average trace of a Hessian matrix of the to-be-quantized object and a loss value before and after quantization of the to-be-quantized object.

12. The system of claim 10 or 11, wherein, The system further comprises an evaluation unit and a rollback unit, The evaluation unit is configured to perform quantization evaluation on the first quantization strategy, and obtain a quantization evaluation result. The rollback unit is configured to, in a case where the quantization evaluation result is unqualified, perform quantization rollback on the first quantization strategy, modify a quantization data type of a to-be-quantized object in the first quantization strategy, and obtain a second quantization strategy, which is used to quantize the one or more to-be-quantized objects.

13. The system of claim 12, wherein, The system further comprises a quantization unit, The quantization unit is configured to quantize the to-be-quantized model using the first quantization strategy or the second quantization strategy, and obtain a quantized model. The quantization unit is configured to send the quantized model to an inference system.

14. The system of claim 12 or 13, wherein, The to-be-quantized model comprises a plurality of network layers, in the first quantization strategy, a quantization data type of a same to-be-quantized object in each network layer is the same, and in the second quantization strategy, the quantization data type of the same to-be-quantized object in each network layer is the same or different, and a number of to-be-quantized objects in the second quantization strategy is less than a number of to-be-quantized objects in the first quantization strategy.

15. The system of claim 14, wherein The rollback unit is configured to obtain a plurality of rollback schemes, each rollback scheme comprising at least one to-be-quantized object, and different rollback schemes comprising different numbers of to-be-quantized objects. The rollback unit is configured to determine a quantization data type of each to-be-quantized object in a first rollback scheme in the plurality of rollback schemes, and obtain a third quantization strategy corresponding to the first rollback scheme. The rollback unit is configured to perform quantization evaluation on the third quantization strategy, and obtain a rollback evaluation result. The rollback unit is configured to, in a case where the rollback evaluation result is qualified, determine the third quantization strategy as the second quantization strategy. The rollback unit is configured to, in a case where the rollback evaluation result is unqualified, determine a quantization data type of each to-be-quantized object in a second rollback scheme, and obtain a fourth quantization strategy corresponding to the second rollback scheme, wherein a number of to-be-quantized objects in the first rollback scheme is not less than a number of to-be-quantized objects in the second rollback scheme.

16. The system of claim 15, wherein The rollback unit is configured to obtain, based on to-be-quantized objects comprised in the first rollback scheme and data types quantifiable by each to-be-quantized object, at least one hierarchical quantization strategy of each network layer in the first rollback scheme. The rollback unit is configured to determine a sensitivity of each hierarchical quantization strategy of each network layer, and determine a quantization strategy of each network layer based on the sensitivity of each hierarchical quantization strategy. The rollback unit is configured to obtain a third quantization strategy corresponding to the first rollback scheme based on the quantization strategy of each network layer.

17. The system of any of claims 10 to 16, wherein, The objects to be quantized include linear transformation objects to be quantized, key-value cache (KV cache) objects to be quantized, set communication objects to be quantized, and attention calculation objects to be quantized. The linear transformation objects to be quantized include an operator performing linear transformation on data, activation value parameters, and weight parameters. The KV cache objects to be quantized include K vectors and V vectors in the KV cache. The attention calculation objects to be quantized include an operator performing matrix multiplication calculation in an attention layer. The set communication objects to be quantized include communication data during set communication.

18. The system of any of claims 10 to 17, wherein, The system further includes a prompting unit, The prompting unit is configured to analyze the rollback evaluation results corresponding to each rollback scheme when the rollback evaluation results of all the rollback schemes are unqualified, to obtain an analysis result, and the analysis result includes objects to be quantized and network layer information whose influence on quantization error is greater than a threshold. The prompting unit is configured to send a prompt message of manual rollback to the client, and the prompt message includes one or more of the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and the analysis result.

19. A computing device, comprising: The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device performs the operation steps of the method according to any one of claims 1 to 8.

20. A cluster of computing devices, characterized in that, The computing device cluster includes at least one computing device, and each computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method according to any one of claims 1 to 9.

21. A chip, characterized by The chip includes a power supply unit and a processing unit. The power supply unit is configured to supply power to the processing unit, so that the processing unit implements the operation steps of the method according to any one of claims 1 to 9.

22. A computer-readable storage medium, characterized in that, The computer program instructions are executed by the computing device, and the computing device performs the operation steps of the method according to any one of claims 1 to 9.

23. A computer program product comprising instructions, characterized in that, The instructions are executed by the computing device, and the computing device performs the operation steps of the method according to any one of claims 1 to 9.

Images