Method for determining quantization strategy, model quantization system and related equipment

Abstract

The invention provides a method for determining a quantization strategy, a model quantization system and related equipment, and the method comprises the following steps: carrying out the free combination of a to-be-quantized object supported by a model and a data type which can be quantized, obtaining a plurality of different quantization strategies, determining the sensitivities of different quantization strategies based on a sensitivity algorithm, and carrying out the quantification of the to-be-quantized object. The sensitivity is used for indicating the influence of the quantization strategies on the precision of the to-be-quantized model, determining a first quantization strategy according to the sensitivity of each quantization strategy, and if the quantization evaluation of the first quantization strategy is qualified, using the first quantization strategy to quantize the model, so that the whole quantization process does not need quantization search; and if the quantization evaluation is unqualified, the precision of part of the to-be-quantized objects in the first quantization strategy is improved in a quantization rollback mode until the second quantization strategy which is evaluated to be qualified is obtained, so that compared with a traditional mode of performing quantization search based on large-scale parameters, the search space can be reduced, and the model quantization efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to a method for determining a quantization strategy, a model quantization system, and related equipment. Background Technology

[0002] With the continuous development of technology, large language models (LLMs) are becoming increasingly larger, and their inference latency is also increasing. To reduce storage overhead and inference latency, model quantization techniques have emerged. Model quantization refers to the process of converting high-precision parameters in a large language model into low-precision parameters, or converting high-precision calculations into low-precision calculations. By reducing the precision of model parameters or calculations, the storage requirements and computational complexity of large language models can be reduced, thereby lowering storage overhead and inference latency.

[0003] However, model quantization techniques alter the precision of model parameters and computation, leading to precision loss in large language models. This precision loss varies across different large language models, necessitating a tailored quantization strategy for each model to ensure the performance degradation after quantization remains within acceptable limits. Typically, a search algorithm is used to determine the optimal quantization strategy for the large language model before performing the quantization operation. However, due to the large number of parameters and computational demands of large language models, the search space for search algorithms is enormous, and the search time increases exponentially with the search space, resulting in low efficiency for large language model quantization. Summary of the Invention

[0004] This application provides a method for determining a quantization strategy, a model quantization system, and related equipment to solve the problem of low model quantization efficiency caused by a large search space and long search time for model quantization strategies.

[0005] Firstly, a method for determining a quantization strategy is provided. This method is applied to a model quantization system and includes the following steps: obtaining one or more objects to be quantized from the model to be quantized, and at least one data type that each object to be quantized can be quantized; determining multiple quantization strategies based on the at least one data type that each object to be quantized can be quantized, wherein each quantization strategy includes a quantization data type for each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can be quantized; determining the sensitivity of each quantization strategy, wherein the sensitivity is used to indicate the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy; and obtaining a first quantization strategy from the multiple quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize one or more objects to be quantized.

[0006] The method described in the first aspect involves freely combining the objects to be quantized and the data types that can be quantized in the model to be quantized, resulting in multiple different quantization strategies. Each quantization strategy has a different data type corresponding to the objects to be quantized. Then, the sensitivity of different quantization strategies is determined based on a sensitivity algorithm. This sensitivity is used to indicate the impact of the quantization strategy on the accuracy of the model to be quantized. The first quantization strategy is determined based on the sensitivity of each quantization strategy, so that the entire quantization process does not require quantization search, thereby solving the problem of low quantization efficiency caused by the long time consumption of quantization search in model quantization.

[0007] In one possible implementation, when determining the sensitivity of each quantization strategy, the sensitivity of each object to be quantized in each quantization strategy can be determined, and the sensitivity of each quantization strategy can be obtained based on the sensitivity of each object to be quantized. The sensitivity of each object to be quantized is obtained based on one or more parameters, such as the average trace of the Hessian matrix of the object to be quantized and the loss value of the object before and after quantization.

[0008] It should be understood that the smaller the average trace of the Hessian matrix of the weights, the greater the accuracy loss caused by the quantization of those weights to the entire model. Therefore, using the average trace as a sensitivity parameter can effectively indicate the impact of the quantization strategy on the model's accuracy. Each object to be quantized can be understood as either a weight parameter or an activation value being quantized during quantization, and the corresponding sensitivity can be calculated according to this principle. Based on the sensitivity of each object to be quantized, the sensitivity of the entire quantization strategy can be obtained.

[0009] The above implementation determines the sensitivity of the entire quantization strategy by calculating the sensitivity of each object to be quantized. Then, the best first quantization scheme is selected based on the sensitivity of the entire quantization strategy. By evaluating accuracy in units of quantization strategies, the problem of the model's accuracy exceeding the controllable range due to the accumulation of accuracy when quantizing multiple objects to be quantized can be avoided. This allows the entire model to quantize objects to be quantized in multiple calculation stages simultaneously, maximizing the quantization degree of the model and developing the overall quantization potential of the model.

[0010] In one possible implementation, after obtaining the first quantization scheme, the first quantization strategy can be evaluated to obtain the quantization evaluation result. If the quantization evaluation result is unqualified, the first quantization strategy can be rolled back to modify the quantization data type of the object to be quantized in the first quantization strategy to obtain the second quantization strategy. The second quantization strategy is used to quantize one or more objects to be quantized.

[0011] The above implementation method, by evaluating the first quantization strategy, determines it as the quantization strategy for the model to be quantized if the evaluation is satisfactory. This allows obtaining a satisfactory quantization strategy without performing a quantization search, while ensuring the efficiency and accuracy of strategy determination. If the evaluation is unsatisfactory, quantization backtracking is performed based on the first quantization strategy, improving the accuracy of some objects to be quantized within the first quantization strategy until a satisfactory second quantization strategy is obtained. Compared to the traditional method of quantization search based on large-scale parameters, determining the first quantization strategy first and then backtracking reduces the search space, thus solving the problem of excessively large search space and long search time during model quantization, which leads to low model quantization efficiency.

[0012] In one possible implementation, the method further includes the following steps: quantizing the model to be quantized using a first quantization strategy or a second quantization strategy to obtain a quantized model, and sending the quantized model to the inference system.

[0013] It should be noted that if the quantitative evaluation result is qualified, the first quantitative strategy can be used to quantize the model; if the quantitative evaluation result is unqualified, the first quantitative strategy is quantized back to obtain the second quantitative strategy in the manner described above, and then the second quantitative strategy is used to quantize the model.

[0014] Furthermore, after determining the second quantization strategy, the model quantization system can feed the solution back to the client. The user can choose whether to use the solution to quantize the model, or the model quantization system can directly use the second quantization strategy to quantize the model, obtain the quantized model data, and then export the quantization results to the client. The user can choose whether to deploy the model data in the inference system, or the model quantization system can use the second quantization strategy to quantize the model, obtain the quantized model data, and then automatically deploy the model data in the inference system, simplifying the user's operation process. The specifics can be determined according to actual business needs, and this application does not impose specific limitations.

[0015] In one possible implementation, the model to be quantized includes multiple network layers. In the first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. In the second quantization strategy, the quantization data type of the same object to be quantized is the same or different in each network layer. The number of objects to be quantized in the second quantization strategy is less than the number of objects to be quantized in the first quantization strategy.

[0016] The above implementation first determines the sensitivity of different quantization strategies, coarsely filtering out a superior first quantization strategy. In this first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. When the first quantization strategy is determined to be unqualified after quantization evaluation, quantization backtracking is performed for each network layer to obtain a more fine-grained second quantization strategy. Compared with the traditional method of sequentially searching for the corresponding quantization data type for a large number of parameters, this method of coarse-grained filtering followed by fine-grained backtracking can reduce the search space, thereby improving the efficiency of quantization for large language models.

[0017] In one possible implementation, when performing quantization backoff on the first quantization strategy to obtain the second quantization strategy, multiple backoff schemes can be obtained first. Each backoff scheme includes at least one object to be quantized, and the number of objects to be quantized varies among different backoff schemes. The quantization data type of each object to be quantized in the first backoff scheme is determined, and the third quantization strategy corresponding to the first backoff scheme is obtained. The third quantization strategy is then quantized to obtain a backoff evaluation result. If the backoff evaluation result is satisfactory, the third quantization strategy is determined as the second quantization strategy. If the backoff evaluation result is unsatisfactory, the quantization data type of each object to be quantized in the second backoff scheme is determined, and the fourth quantization strategy corresponding to the second backoff scheme is obtained. The number of objects to be quantized in the first backoff scheme is not less than the number of objects to be quantized in the second backoff scheme.

[0018] For example, rollback plan 1 includes object 1 and object 2 to be quantified; rollback plan 2 includes object 1, object 2, and object 3 to be quantified; rollback plan 3 includes object 1 and object 3 to be quantified, and so on. Then, following the search order, the optimal quantification strategy under each rollback plan is determined sequentially, and the optimal quantification strategy is quantitatively evaluated. If the evaluation result is satisfactory, the quantification strategy is determined as the second quantification strategy; if it is unsatisfactory, the optimal quantification strategy under the next rollback plan is determined, and so on, until a second quantification strategy with satisfactory evaluation results is obtained.

[0019] The above implementation pre-establishes backoff schemes containing different objects to be quantized. Then, following a search order that gradually reduces the number of objects to be quantized, it determines the optimal third quantization strategy under the first backoff scheme. It then evaluates the strategy, and if it fails, it determines the optimal quantization strategy under the next backoff scheme according to the search order, until a satisfactory quantization strategy is obtained. In this way, if there are M backoff schemes, it only needs to be evaluated M times at most to obtain a satisfactory second quantization strategy, thereby further improving the efficiency of quantization backoff.

[0020] In one possible implementation, when determining the quantization data type of each object to be quantized in the first backoff scheme among multiple backoff schemes, and obtaining the third quantization strategy corresponding to the first backoff scheme, it is possible to obtain at least one level quantization strategy for each network layer under the first backoff scheme based on the objects to be quantized included in the first backoff scheme and at least one data type that each object to be quantized can be quantized, determine the sensitivity of each level quantization strategy of each network layer, determine the quantization strategy of each network layer based on the sensitivity of each level quantization strategy, and obtain the third quantization strategy corresponding to the first backoff scheme based on the quantization strategy of each network layer.

[0021] The above implementation, when determining the optimal third quantization strategy for each backoff scheme, still suffers from a large search space issue when using traditional quantization search because it requires a more granular determination of the quantization strategy for each network layer. Using sensitivity-based determination can more quickly obtain the quantization strategy for each network layer, thereby improving the efficiency of quantization backoff. Compared to solutions that directly search the massive search space across multiple network layers, multiple objects to be quantized, and multiple data types, the technical solution of this application can obtain the optimal quantization strategy more efficiently and with higher quality, thereby improving the quantization efficiency of the model.

[0022] In one possible implementation, the objects to be quantized include linear transformation objects, key-value cache (KVcache) objects, set communication objects, and attention computation objects. The linear transformation objects include operators that perform linear transformations on the data, activation parameters, and weight parameters. The KVcache objects include K and V vectors in the KVcache. The attention computation objects include operators that perform matrix multiplication in the attention layer. The set communication objects include communication data during set communication.

[0023] The above implementation maximizes the quantization level of the model by simultaneously quantizing multiple objects to be quantized, thus unlocking the overall quantization potential of the model. Furthermore, by selecting the optimal first quantization scheme based on the sensitivity of the entire quantization strategy, the problem of accumulated precision leading to uncontrollable model accuracy issues caused by quantizing multiple objects can be avoided.

[0024] In one possible implementation, the method further includes the following steps: if all the backoff evaluation results of each backoff scheme are unqualified, analyze the backoff evaluation results of each backoff scheme to obtain analysis results, the analysis results include the quantized object and network layer information whose impact on quantization error is greater than the threshold, and send a manual backoff prompt message to the client held by the user, the prompt message including one or more of the third quantization strategy corresponding to each backoff scheme, the backoff evaluation results corresponding to each third quantization strategy, and the analysis results.

[0025] In practical implementation, when prompting the user to manually roll back, the message can include the optimal quantization strategy and quantization evaluation results for each rollback option. Although these quantization strategy evaluation results are unsatisfactory, they can still be used as a reference for the user, facilitating a faster acquisition of a quantization strategy for manual rollback. Furthermore, the quantization evaluation results of the optimal quantization strategy for each rollback option can be analyzed, and the analysis results can be displayed to the user, such as showing the mean squared error of each quantization strategy. This helps the user determine which objects to be quantized and which network layers have larger errors, allowing the user to modify the quantization data type of the objects to be quantized and obtain a second quantization strategy.

[0026] If the user requires a high model accuracy, and the model's performance cannot meet the user's needs after quantization rollback, the user can be asked to provide a custom quantization strategy. To help the user determine the quantization parameters, the network layers and parameters that have a significant impact on performance can be identified based on the previous quantization evaluation results and rollback evaluation results. This allows the user to customize the quantization strategy and further improves the user experience.

[0027] Secondly, a model quantization system is provided, comprising: an acquisition unit for acquiring one or more objects to be quantized from the model to be quantized, and at least one data type that each object to be quantized can be quantized; a combination unit for determining multiple quantization strategies based on the at least one data type that each object to be quantized can be quantized, wherein each quantization strategy includes a quantization data type for each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can be quantized; a determination unit for determining the sensitivity of each quantization strategy, wherein the sensitivity is used to indicate the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy; and a determination unit for acquiring a first quantization strategy from the multiple quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize one or more objects to be quantized.

[0028] The system described in the second aspect is implemented in which the quantized objects of the model to be quantized and the data types that can be quantized are freely combined to obtain a variety of different quantization strategies. The data types corresponding to the objects to be quantized in each quantization strategy are different. Then, the sensitivity of different quantization strategies is determined based on a sensitivity algorithm. This sensitivity is used to indicate the impact of the quantization strategy on the accuracy of the model to be quantized. The first quantization strategy is determined according to the sensitivity of each quantization strategy, so that the entire quantization process does not require quantization search, thereby solving the problem of low quantization efficiency caused by the long time consumption of quantization search in model quantization.

[0029] In one possible implementation, a determining unit is used to determine the sensitivity of each object to be quantized in each quantization strategy, and to obtain the sensitivity of each quantization strategy based on the sensitivity of each object to be quantized, wherein the sensitivity of each object to be quantized is obtained based on one or more parameters among the mean trace of the Hessian matrix of the object to be quantized and the loss value of the object to be quantized before and after quantization.

[0030] In one possible implementation, the system further includes an evaluation unit and a backoff unit. The evaluation unit is used to perform quantitative evaluation on the first quantization strategy and obtain the quantitative evaluation result. The backoff unit is used to perform quantitative backoff on the first quantization strategy when the quantitative evaluation result is unqualified, modify the quantization data type of the object to be quantized in the first quantization strategy, and obtain a second quantization strategy. The second quantization strategy is used to quantize one or more objects to be quantized.

[0031] In one possible implementation, the system further includes a quantization unit, which is used to quantize the model to be quantized using a first quantization strategy or a second quantization strategy to obtain a quantized model, and the quantization unit is used to send the quantized model to the inference system.

[0032] In one possible implementation, the model to be quantized includes multiple network layers. In the first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. In the second quantization strategy, the quantization data type of the same object to be quantized is the same or different in each network layer. The number of objects to be quantized in the second quantization strategy is less than the number of objects to be quantized in the first quantization strategy.

[0033] In one possible implementation, a rollback unit is used to acquire multiple rollback schemes, each rollback scheme including at least one object to be quantized, and the number of objects to be quantized varies among different rollback schemes. The rollback unit is also used to determine the quantization data type of each object to be quantized in the first rollback scheme among the multiple rollback schemes, obtaining a third quantization strategy corresponding to the first rollback scheme. Furthermore, the rollback unit is used to perform a quantization evaluation on the third quantization strategy to obtain a rollback evaluation result. If the rollback evaluation result is satisfactory, the rollback unit is used to determine the third quantization strategy as a second quantization strategy. If the rollback evaluation result is unsatisfactory, the rollback unit is used to determine the quantization data type of each object to be quantized in the second rollback scheme, obtaining a fourth quantization strategy corresponding to the second rollback scheme. The number of objects to be quantized in the first rollback scheme is not less than the number of objects to be quantized in the second rollback scheme.

[0034] In one possible implementation, the backoff unit is used to obtain at least one level quantization strategy for each network layer under the first backoff scheme based on the objects to be quantized included in the first backoff scheme and the data type that each object to be quantized can be quantized; the backoff unit is used to determine the sensitivity of each level quantization strategy of each network layer; based on the sensitivity of each level quantization strategy, determine the quantization strategy of each network layer; and the backoff unit is used to obtain a third quantization strategy corresponding to the first backoff scheme based on the quantization strategy of each network layer.

[0035] In one possible implementation, the objects to be quantized include linear transformation objects, key-value cache (KVcache) objects, set communication objects, and attention computation objects. The linear transformation objects include operators that perform linear transformations on the data, activation parameters, and weight parameters. The KVcache objects include K and V vectors in the KVcache. The attention computation objects include operators that perform matrix multiplication in the attention layer. The set communication objects include communication data during set communication.

[0036] In one possible implementation, the system further includes a prompting unit, which is used to analyze the back-off evaluation results corresponding to each back-off scheme when all back-off evaluation results of each back-off scheme are unqualified, and obtain analysis results. The analysis results include the quantized objects and network layer information whose impact on quantization error is greater than the threshold. The prompting unit is used to send a prompt message for manual back-off to the client. The prompt message includes one or more of the following: the third quantization strategy corresponding to each back-off scheme, the back-off evaluation results corresponding to each third quantization strategy, and the analysis results.

[0037] Thirdly, a computing device is provided, the computing device including a processor and a memory, the memory for storing instructions and the processor for executing the instructions, such that the computing device implements the method described in the first aspect.

[0038] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0039] Fifthly, a computing device cluster is provided, the computing device cluster including at least one computing device, each computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster implements the method described in the first aspect.

[0040] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product including instructions capable of running on a computing device or stored in any available medium, and when the computer program product is run on a computing device or a cluster of computing devices, causing the computing device or cluster of computing devices to perform the method described in the first aspect.

[0041] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0042] Figure 1 This is an architecture diagram of a model quantization system provided in this application;

[0043] Figure 2 This is an example diagram of a model quantization system deployed in a cloud environment, as provided in this application;

[0044] Figure 3 This is a flowchart illustrating the steps of a method for determining a quantification strategy provided in this application;

[0045] Figure 4 This is an example diagram of the object to be quantized in a method for determining a quantization strategy provided in this application;

[0046] Figure 5 This is a schematic diagram of the structure of the model quantization system provided in this application;

[0047] Figure 6 This is a schematic diagram of the structure of a computing device provided in this application;

[0048] Figure 7 This is an example diagram of a computing device cluster provided in this application;

[0049] Figure 8 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0050] First, the application scenarios involved in this application will be explained.

[0051] With the continuous development of technology, large language models (LLMs) are becoming increasingly larger, and their inference latency is also increasing. In order to reduce the storage overhead and inference latency of the models, model quantization technology has emerged. Model quantization refers to the process of converting high-precision parameters in a large language model into low-precision parameters, or high-precision computation into low-precision computation. Converting parameter precision can be called parameter quantization, which involves converting the parameters to be quantized in the model (such as weights) from high-precision data types to low-precision data types, such as from floating-point to integer types, thereby reducing the storage space occupied by the model. Converting computation precision can be called operator quantization, which involves converting the operators to be quantized in the model (such as convolution, activation functions, matrix multiplication, etc.) from high-precision computation to low-precision computation, such as from floating-point operations to integer operations. Unlike parameter quantization, operator quantization focuses on the entire computation process, not just a single parameter. Operator quantization can quantize multiple parameters involved in the computation, such as weights, input data, output data, and activation values. This application does not make specific limitations. Simply put, parameter quantization is mainly used to reduce model size, while operator quantization is mainly used to accelerate computation and reduce computational resources; their focuses differ. By reducing the precision of model parameters or computation, the storage requirements and computational complexity of large language models can be reduced, thereby reducing the storage overhead and inference latency of large language models.

[0052] Model quantization techniques alter model parameters and computational precision, thus introducing accuracy loss into large language models. The degree of accuracy loss varies across different large language models; therefore, each large language model requires a tailored quantization strategy to ensure that the performance degradation after quantization remains within acceptable limits. Typically, before quantization, a search algorithm is used to determine the optimal quantization strategy for the large language model, including the parameters to be quantized and the quantization method, before performing the appropriate quantization operation.

[0053] However, large language models are very large in size and have a large number of parameters, which makes the search space of the search algorithm very large when quantizing large language models. The search time increases exponentially with the size of the search space, resulting in low efficiency of large language model quantization.

[0054] Meanwhile, the reasoning process of large language models is quite complex, involving many different computational stages. The objects to be quantized differ in each computational stage, resulting in varying precision losses. If quantization were performed in every computational stage, the precision losses would accumulate, causing the final model's quantization loss to exceed the user's acceptable range. Therefore, current large language models typically select only one type of object to be quantized, such as quantizing only matrix multiplication (matmul) operators or only quantizing data stored in the KV cache, while maintaining high-precision computation in other computational stages to ensure controllable quantization precision. However, this approach results in some parameters or computations that could reduce precision not being quantized, limiting the overall quantization potential of the large language model and failing to meet user needs in terms of model quantization level.

[0055] In summary, current large language models suffer from low quantization efficiency due to their large size, resulting in an excessively large search space and long search time for quantization strategies. Furthermore, large language models typically only allow quantization for one computational stage; otherwise, they are prone to cumulative accuracy loss. However, quantizing only one stage limits the overall quantization potential of the large language model, leading to low quantization precision. Therefore, current large language models suffer from low quantization efficiency and low quantization precision, making it difficult to meet user needs.

[0056] To address the problem of excessively large search space and long search time during model quantization, leading to low quantization efficiency, this application provides a method for determining quantization strategies. This method involves freely combining the objects to be quantized supported by the model to be quantized and the data types that can be quantized to obtain multiple different quantization strategies. Then, a sensitivity algorithm is used to determine the sensitivity of different quantization strategies, which indicates the impact of the quantization strategy on the accuracy of the model to be quantized. A first quantization strategy is determined based on the sensitivity of each strategy, and then the first quantization strategy is evaluated. If the evaluation is satisfactory, the first quantization strategy is determined as the quantization strategy for the model to be quantized, eliminating the need for quantization search in the entire quantization process and thus solving the problem of low quantization efficiency caused by long quantization search time. If the evaluation is unsatisfactory, quantization backtracking is performed based on the first quantization strategy, improving the accuracy of some objects to be quantized using the first quantization strategy until a satisfactory second quantization strategy is obtained. Compared to the traditional method of quantization search based on large-scale parameters, this method of determining the first quantization strategy first and then backtracking reduces the search space, thus solving the problem of excessively large search space and long search time during model quantization, leading to low model quantization efficiency.

[0057] Figure 1This is an architecture diagram of a model quantization system provided in this application, such as... Figure 1 As shown, the architecture includes a client 100, a model quantization system 200, and an inference system 300. Communication connections exist between the client 100, the model quantization system 200, and the inference system 300. These connections can be wired or wireless networks. Wired network connections include Ethernet, wired lines, cables, etc., while wireless network connections include Wi-Fi, cellular networks, etc. They may also include the Internet, local area networks (LANs), etc., which simultaneously support wired and wireless network connections. Other network types are also possible, and this application does not specify a particular type. The number of clients 100 and inference systems 300 can be one or more. Figure 1 This application uses one client and one inference system as an example for illustration, and does not impose any specific limitations.

[0058] Client 100 is deployed on terminal devices or computing devices to enable human-computer interaction. Terminal devices include personal computers, smartphones, wearable devices, handheld processors, tablets, mobile laptops, augmented reality (AR) devices, virtual reality (VR) devices, smart conferencing devices, etc., without specific limitations. Computing devices can be bare metal servers (BMS), virtual machines, or containers. BMS refers to a general-purpose physical server, such as an ARM server or an x86 server; a virtual machine refers to a complete computer system with full hardware system functionality simulated by software, running in a completely isolated environment. Any task that can be performed on a physical computer can also be performed in a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity needs to be used as the virtual machine's hard drive and memory capacity. Each virtual machine has its own independent basic input / output system (BIOS), hard disk, and operating system, and can be operated like a physical machine. A container is a portable software unit that can combine an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process.

[0059] The model quantization system 200 can be deployed on computing devices or a cluster of computing devices. The description of the computing devices is as described above and will not be repeated here. It can also be deployed on a chip or a chip system composed of multiple chips. The chip may include a processor implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), or any combination thereof; this application does not impose specific limitations. It may also include accelerator cards that have good performance when performing machine learning tasks and can improve the speed and efficiency of neural network training, such as graphics processing units (GPUs), neural network processing units (NPUs), data processing units (DPUs), and tensor processing units (TPUs).

[0060] The inference system 300 can be deployed on the aforementioned terminal devices, computing devices, or computing device clusters, and can also be deployed on inference cards, edge devices, etc., without specific limitations in this application. Among them, the inference card is hardware specifically designed to accelerate the inference process of AI models, and the edge device is hardware device that is close to the data source or data user.

[0061] Optionally, the client 100 and the model quantization system 200 may be deployed on the same computing device, or the client 100 and the model quantization system 200 may be deployed on different computing devices in the same computing device cluster, or the client 100 and the model quantization system 200 may be deployed on different computing devices in different computing device clusters. This application does not make any specific limitations.

[0062] Optionally, the inference system 300 and the model quantization system 200 may be deployed on the same computing device, or the inference system 300 and the model quantization system 200 may be deployed on different computing devices in the same computing device cluster, or the inference system 300 and the model quantization system 200 may be deployed on different computing devices in different computing device clusters. This application does not make any specific limitations.

[0063] Optionally, client 100 can be software or an application running on a user-controlled terminal device or computing device, such as a personal computer (PC) client, a web client accessed via a browser, an application (APP) client running on a mobile terminal, or a cloud platform console; this application does not impose specific limitations. The user holding client 100 can be a developer of AI models, such as a programmer or AI engineer; this application does not impose specific limitations.

[0064] Optionally, the client 100 can be a model quantization tool. Such tools are specifically designed for model quantization. Users can use this tool to quantize the model before deploying it to the production environment (i.e., the inference system 300), reducing the precision of parameters or calculations in the model, thereby lowering memory requirements and computational resource consumption. The model can then be deployed to the production environment, allowing it to run on lighter devices, such as smartphones and edge devices, while maintaining performance. This model quantization tool can achieve model quantization using the method for determining the quantization strategy provided in this application.

[0065] Optionally, the client 100 may be a model service platform or a client of a machine learning framework for handling machine learning-related tasks. Such clients provide a series of tools and services for the entire process of defining, training, deploying, inferring, and managing machine learning models. These clients include the model quantization function provided in this application to realize the quantization and compression of models.

[0066] Optionally, client 100 can also be a client of a cloud platform, used for users to purchase and rent various cloud services. The method for determining the quantization strategy provided in this application can be one of the cloud services, which users can purchase separately; or, the cloud platform provides users with a comprehensive service, and the method for determining the quantization strategy provided in this application can be a sub-service of the comprehensive cloud service. For example, if the comprehensive cloud service is an AI cloud service, the method for determining the quantization strategy provided in this application can be a sub-service of the AI ​​cloud service. After the model is trained but before it is deployed to the generation environment, this sub-service can be used to quantize and compress the model.

[0067] The preceding text described in detail the possible deployment methods for client 100, model quantization system 200, and inference system 300. In actual deployment, flexible deployment can be carried out based on specific application scenarios and business requirements. The following section provides examples of specific deployment methods for client 100, model quantization system 200, and inference system 300.

[0068] For example, client 100, model quantization system 200, and inference system 300 can be deployed on the enterprise's internal network. Model quantization system 200 and inference system 300 are deployed on servers or server clusters purchased by the enterprise. Client 100 is a remote desktop connection tool or server management console software. Employees of the enterprise can remotely connect to the server or server cluster where model quantization system 200 is located through client 100, use model quantization system 200 to quantize and compress the trained model, and the quantized and compressed model can be deployed on the server or server cluster where inference system 300 is located.

[0069] For example, client 100, model quantization system 200, and inference system 300 can be deployed in a cloud environment. Figure 2 This is an example diagram of a model quantization system deployed in a cloud environment, as provided in this application. Figure 2 As shown, a user can initiate a purchase request for AI cloud services through client 100. These AI cloud services include sub-services corresponding to the method for determining the quantization strategy provided in this application. Specifically, after client 100 sends the purchase request to the cloud platform, the cloud platform can grant client 100 access to the model quantization system 200 provided in this application. This allows the user to use the model quantization system 200 to quantize and compress the model through client 100. After obtaining the quantized and compressed lightweight model, the user can deploy the lightweight model in the inference system 300.

[0070] The cloud platform also maintains various basic resources, including computing resources, storage resources, network resources, and security resources, to meet the computing needs of the model quantization system 200 under different scales and loads. Furthermore, these computing resources can be dynamically scaled according to the usage requirements of the model quantization system 200 to ensure the stable operation of the model quantization system 200 and provide users with reliable model quantization services.

[0071] Optionally, the inference system 300 and the model quantization system 200 can be services provided by the same cloud platform. Figure 2 The cloud platform shown includes an inference system 300 and a model quantization system 200. Alternatively, the inference system 300 and the model quantization system 200 can be services provided by different cloud platforms. In this case, a hybrid cloud architecture can be used to achieve data communication between the inference system 300 and the model quantization system 200. That is, the model quantization system 200 is deployed in data center A, and the inference system 300 is deployed in data center B. Data center A and data center B communicate with each other through their respective cloud platforms.

[0072] It is worth noting that, Figure 2 In the example shown, the model quantization system 200 is deployed and implemented in a cloud environment. Figure 2 This is merely an exemplary description of the model quantization system 200 involved in this application, and this application does not impose any specific limitations.

[0073] Alternatively, the model quantization system 200 can also be deployed in multiple cloud environments, for example, Figure 1 The confirmation process of the first quantization strategy shown is implemented in the first cloud environment, the process of quantizing and evaluating the first quantization strategy to obtain the quantization evaluation result is implemented in the second cloud environment, the quantization rollback is implemented in the third cloud environment, and the final model quantization result is obtained in the fourth cloud environment. The above examples are for illustration only and are not specifically limited in this application.

[0074] Alternatively, the model quantization system 200 can also be implemented through deployment in multiple non-cloud environments, for example, Figure 1 The confirmation process of the first quantification strategy shown is implemented on the internal office server of Company A. The process of quantifying and evaluating the first quantification strategy to obtain the quantification evaluation results and the quantification rollback process are implemented on the internal office server of Company B. The final model quantification results are implemented on the internal office server of Company C. Companies A, B and C have a cooperative relationship. The above example is for illustration and is not intended to limit the specific scope of this application.

[0075] Optionally, the model quantization system 200 can also be deployed in multiple cloud and non-cloud environments. For example, Figure 1The confirmation process of the first quantification strategy shown is implemented on the internal office server of Company A. The process of quantifying and evaluating the first quantification strategy to obtain the quantification evaluation results and the process of quantification rollback are implemented in the first cloud environment. The final model quantification results are implemented on the internal office server of Company B. The above example is for illustration and is not specifically limited in this application.

[0076] It should be understood that the above description, in conjunction with specific scenarios, exemplifies some possible deployment forms of the model quantization system 200. In actual implementation, other deployment forms may exist, and this application does not limit the deployment method of the model quantization system 200. For ease of description, the following embodiments use... Figure 1 The system structure shown is used as an example for illustration.

[0077] It should be understood that the model quantized in this application can be any model in the field of AI. As can be seen from the foregoing, the more model parameters there are, the worse the quantization effect. The large language model is currently the largest model in terms of scale and number of parameters. For ease of description, the following embodiments will be illustrated using the large language model as an example.

[0078] In this embodiment, the model quantization system 200 can obtain the model to be quantized from the client 100, confirm that the model supports at least one quantized object, and that each quantized object can be quantized by at least one data type. Multiple quantization strategies are generated through permutations and combinations, each strategy including the quantized data type of each quantized object, which is one of the aforementioned at least one data type. Then, the sensitivity corresponding to each quantization strategy is calculated, indicating the degree of accuracy loss. Based on the sensitivity of each quantization strategy, a first quantization strategy with the lowest accuracy loss is obtained. The first quantization strategy is then evaluated. If the evaluation result is satisfactory, the first quantization strategy is used to quantize the model to be quantized, obtaining the model quantization result. If the evaluation result is unsatisfactory, the first quantization strategy is rolled back, increasing the accuracy of some quantized data types of the quantized objects until the evaluation result is satisfactory, obtaining a second quantization strategy. The second quantization strategy is then used to quantize the model to be quantized, obtaining the model quantization result. Finally, the model quantization result is sent to the inference system 300 for production.

[0079] For example, with Figure 1 For example, after receiving the model to be quantized sent by the client 100, the model quantization system 200 determines the various objects to be quantized that can participate in the quantization of the model. Figure 1 Taking four objects to be quantized as an example, we then determine at least one data type that each object can be quantized. Figure 1Taking the example of each object to be quantized having two data types, and then arranging and combining them, we can obtain 2 4 =16 permutations and combinations, which means 16 quantitative strategies ( Figure 1 Two examples are shown. Then, the sensitivity of each quantization strategy is calculated. Assuming that the lower the sensitivity, the smaller the accuracy loss, the scheme with the lowest sensitivity can be selected as the first quantization strategy. Figure 1 In the example, assuming quantization strategy 2 is the first quantization strategy, quantization strategy 2 can be evaluated. If it is qualified, the model will be quantized according to this scheme to obtain the model quantization result. If it is not qualified, quantization strategy 2 will be rolled back. For example, if the precision of data type 1 is lower than that of data type 2, the data type 2 of the object to be quantized, 4, can be changed to data type 1. Then, it can be determined whether the model precision after this rollback meets the requirements. If it does not meet the requirements, the data type 2 of the object to be quantized, 3, can be changed to data type 1, and the model precision after this rollback can be determined. This process continues until the model precision meets the requirements, obtaining the final quantization strategy. Then, this scheme is used to quantize the model to obtain the model quantization result.

[0080] Optionally, the object to be quantized refers to an object in the model that can be quantized. Here, "object to be quantized" is a category, and each object to be quantized can include one or more quantization parameters or quantization operators. Parameters in the model can be classified according to their function. Each network layer may include many parameters of the same type. Some types of parameters can be quantized, while others cannot. For example, some dynamic parameters, if quantized, would require dynamic adjustment, making quantization difficult; these types of parameters generally do not participate in quantization. Similarly, some specific activation functions, such as the ReLU function, may lead to information loss after quantization; these types of parameters generally do not participate in quantization. Small parameter values, such as those close to 0, may lead to a loss of numerical precision; these types of parameters generally do not participate in quantization. Of course, there are other parameters that cannot participate in quantization, which will not be listed here. It should be understood that the object to be quantized in this application refers to a certain type of parameter that can participate in quantization.

[0081] For example, the objects to be quantized may include linear transformation objects, KV cache objects, attention computation objects, and ensemble communication objects. The linear transformation objects include operators that perform linear transformations on the data, as well as activation value parameters and weight parameters that participate in the linear transformation. The KV cache objects include K vectors and V vectors in the KV cache. The attention computation objects include operators that perform matrix multiplication in the attention layer. The ensemble communication objects include communication data during ensemble communication.

[0082] Optionally, data type refers to the numerical format of data in the model. Data types include floating-point types (float) and integer types (linteger). Floating-point data types include float16 (fp16), float32 (fp32), etc., where fp16 represents a 16-bit floating-point format, consisting of 1 sign bit, 5 exponent bits, and 10 mantissa bits; fp32 represents a 32-bit floating-point format, consisting of 1 sign bit, 8 exponent bits, and 23 mantissa bits. The larger the number, the greater the numerical precision, and the greater the corresponding storage and computing resource requirements. Integer data types include int8, int4, etc., where int8 represents an 8-bit integer format, representing a range from -128 to 127, or from 0 to 255 (unsigned); int4 represents a 4-bit integer format, representing a range from -8 to 7, or from 0 to 15 (unsigned). Understandably, floating-point data types have higher precision and therefore require more storage and computing resources, while integer data types have lower precision and therefore require less storage and computing resources.

[0083] Optionally, the data type that the object to be quantized can be quantized includes the data types that the object can use. Typically, the data types that can be used by different objects to be quantized vary depending on the model structure, hardware platform, computational efficiency, and task requirements. For example, in high-precision tasks such as medical image processing and financial forecasting, higher bit-width data types, such as fp16 and fp32, are required. For scenarios that allow for a certain degree of error, such as object detection or image classification tasks, lower bit-width data types, such as int8 and int4, can be used. Different hardware platforms also support different data types. For example, some CPUs and GPUs support int8 operations, while some TPUs may have special optimizations for fp16. Some embedded or edge devices, due to limited resources, can only support lower bit-width data types, such as int8 or int4. Therefore, it is necessary to determine the data type that can be used by the object to be quantized based on the specific scenario.

[0084] Furthermore, as mentioned above, quantization refers to converting a model from high precision to low precision. Therefore, the precision of the data type that can be quantized is less than the current data type. For example, if the object to be quantized is currently in fp16 floating-point format, then the data type that can be quantized can only be a data type with lower precision than fp16 (such as integer types int8 or int4), not a data type with higher precision than fp16 (such as fp32). The above examples are for illustrative purposes only and are not intended to impose specific limitations in this application.

[0085] Optionally, the sensitivity of each quantization strategy can be obtained based on one or more sensitive parameters related to model accuracy. Specifically, after calculating one or more of the aforementioned sensitive parameters, the sensitivity can be determined by weighted summation or product of these sensitive parameters. The sensitive parameters may include the average trace of the Hessian matrix of the object to be quantized, the loss function values ​​of the object before and after quantization, and may include more content, which can be determined according to the actual business scenario; this application does not impose specific limitations. The Hessian matrix is ​​a matrix composed of the second-order partial derivatives of a multivariate function, typically used to describe the local curvature of a function. This matrix can help determine the extreme case of a function at a certain point and the nature of this extreme point, which can be a maximum, minimum, or saddle point. The average trace of the Hessian matrix refers to the sum of the diagonal elements of the matrix.

[0086] For example, for the object to be quantized by linear transformation, the sensitivity can be obtained based on the product of the average trace of the Hessian matrices of the linear layer parameters and the loss function value. The linear layer parameters include activation values ​​and weights. For the object to be quantized by KV cache, the sensitivity can be obtained based on the average trace of the Hessian matrices of the K and V vectors stored in the KV cache. For the object to be quantized by attention computation, the sensitivity can be obtained based on the average trace of the Hessian matrices of the Q, K, and V vectors involved in the attention computation. For the object to be quantized by ensemble communication, the sensitivity can be obtained based on the average trace of the Hessian matrices of the data to be communicated participating in the ensemble communication. The above examples are for illustrative purposes only and are not intended to be specific limitations in this application.

[0087] It should be understood that the model quantization system 200 of this application first determines the quantizable objects in the model and the data types that each quantizable object can be quantized. Then, it determines the quantization strategy of the model through permutation and combination. The quantization strategy includes the quantization data type corresponding to each quantizable object, which is one of the aforementioned quantizable data types. Then, it determines the sensitivity value of each quantization strategy. By evaluating accuracy on a per-strategy basis, it avoids the problem of accuracy accumulation leading to the model accuracy exceeding the controllable range due to quantizing multiple quantizable objects. This allows the entire model to quantize objects in multiple computational stages simultaneously, maximizing the quantization degree of the model and developing the overall quantization potential of the model. At the same time, by judging the accuracy loss of the model by the quantization strategy based on sensitivity, it is not necessary to use a test set to quantize and evaluate the model under each quantization strategy. The first quantization strategy can be quickly selected based on sensitivity.

[0088] Furthermore, if the model's performance meets the standards after the first quantization strategy determined based on the sensitivity value passes the quantization evaluation, the first quantization strategy can be directly used as the final solution to quantize the model. This eliminates the need for quantization search in the entire quantization process, and the quantization evaluation only needs to be performed once, thereby solving the problem of low quantization efficiency caused by the long quantization search time in model quantization.

[0089] If the model does not meet the requirements, a second quantization strategy can be searched out by quantization backoff. Compared with the traditional method of sequentially searching for the corresponding quantization data types for a large number of parameters, this method of backoff with the first quantization strategy can reduce the search space and thus improve the efficiency of quantization of large language models.

[0090] For example, suppose the model has 10,000 parameters, and each parameter may have 3 data types, then the search space is 3. 10000 The search space is extremely large, and the computational cost is extremely high. However, using the technical solution of this application, assuming the first quantization strategy is: all parameters use data type 1, and through quantization backoff, the maximum search space is 3. 10000 In other words, every parameter needs its data type adjusted, which is the worst-case scenario. The best-case scenario is if only one parameter needs adjustment, resulting in a backtracking search space of 3. Since the initial backtracking strategy is the first quantization strategy determined through sensitivity analysis, the number of parameters requiring adjustment won't be too large. If only 10% of the parameters need adjustment, then the backtracking search space is 3. 1000 The search space is much smaller than that of traditional techniques. Therefore, using the technical solution of this application can significantly reduce the number of quantization searches. If the performance of the first quantization strategy meets the requirements, quantization searches may not even be necessary, greatly improving the efficiency of model quantization.

[0091] Optionally, the model quantization system 200 can pre-set M backoff schemes, each including different quantization data types corresponding to the objects to be quantized. Then, based on the objects to be quantized and the quantization data types in the backoff scheme, the optimal quantization strategy under that backoff scheme is determined, and then quantization is evaluated. If it passes the evaluation, it is adopted as the second quantization strategy. If it fails, the optimal quantization strategy under the next backoff scheme is determined based on the objects to be quantized and the quantization data types in the next backoff scheme, and so on, until a second quantization strategy that passes the quantization evaluation is obtained. In this way, a second quantization strategy can be obtained with a maximum of M evaluations, which can improve the efficiency of quantization backoff.

[0092] For example, Option 1 includes the linear transformation object to be quantized and the KV cache object to be quantized. This fallback option means that only these two objects to be quantized in the model are quantized, and other objects to be quantized are not quantized. Option 2 includes the linear transformation object to be quantized, the KV cache object to be quantized, and the set communication object. This fallback option means that all three objects to be quantized in the model are quantized, and so on. Examples will not be given here.

[0093] Furthermore, the model quantization system 200 can determine the search order of the quantization strategy based on the number of objects to be quantized in the M backoff schemes. The more objects to be quantized in a backoff scheme, the earlier the search order. In other words, during the search, the number of objects to be quantized participating in the search is gradually reduced, while the number of objects to be quantized during the backoff is gradually increased. For example, if there are 5 objects to be quantized, the search is first performed with 4 objects to be quantized, which means backoffing 1 object. Then, the search is performed with 3 objects to be quantized, which means backoffing 2 objects. Then, the search is performed with 2 objects to be quantized, which means backoffing 3 objects, and so on. By gradually reducing the number of objects to be quantized, large-scale backoffs can be avoided, and backoff efficiency can be improved.

[0094] For example, Figure 1 In the example shown, when performing quantization backoff on quantization strategy 2, the search can first be performed on the objects to be quantized (1, 2, 3). The searched quantization strategies are then evaluated. If they pass the evaluation, this strategy becomes the final quantization strategy. If they fail, the search continues on the objects to be quantized (1, 2, 4), and the search is repeated. This process continues until no suitable quantization strategy is found for all backoff schemes involving three objects. Then, the search begins for backoff schemes involving two objects, and so on, until a suitable quantization strategy is found. The above example is for illustrative purposes only and is not intended to limit the scope of the application.

[0095] Furthermore, when the model quantization system searches for the optimal quantization strategy based on the backoff scheme, it can also use the aforementioned sensitivity calculation method to calculate the optimal quantization strategy. If the object to be quantized includes parameters of multiple network layers, the quantization data type of the corresponding parameters of each network layer can be calculated at the network layer level to obtain a more refined quantization strategy. That is to say, in the first quantization strategy, all parameters of an object to be quantized use a unified quantization data type, such as all activation values ​​and weights of all linear layers using the w8a8 quantization data type. During quantization backoff, the quantization data type of the activation values ​​and weights of each linear layer can be determined, for example, the first network layer is w8a8, the second network layer is w4a8, the third network layer is w4a16, and so on. This application does not make specific limitations.

[0096] It should be understood that during quantization backoff, searching for quantization strategies in a more granular manner and completing the refined search according to a certain backoff order can not only obtain more refined and accurate quantization strategies, but also improve search efficiency. Compared with the approach of searching directly in the massive search space of multiple network layers, multiple objects to be quantized, and multiple data types, the technical solution of this application can obtain the optimal quantization strategy more efficiently and with higher quality, thereby improving the quantization efficiency of the model.

[0097] It should be understood that after determining the second quantization strategy, the model quantization system 200 can feed the solution back to the client 100. The user can choose whether to use the solution to quantize the model. Alternatively, the model quantization system 200 can directly use the second quantization strategy to quantize the model, obtain the quantized model data, and then export the quantization result to the client 100. The user can choose whether to deploy the model data to the inference system 300. Alternatively, the model quantization system 200 can also use the second quantization strategy to quantize the model, obtain the quantized model data, and then automatically deploy the model data to the inference system 300, simplifying the user's operation process. The specific method can be determined according to the actual business needs, and this application does not impose specific limitations.

[0098] In summary, this application provides a model quantization system. This system obtains multiple quantization strategies by freely combining the objects to be quantized supported by the model to be quantized and the data types that can be quantized. Then, a sensitivity algorithm is used to determine the sensitivity of different quantization strategies, which indicates the impact of the quantization strategy on the accuracy of the model to be quantized. A first quantization strategy is determined based on the sensitivity of each strategy, and then the first quantization strategy is evaluated. If the evaluation is satisfactory, the first quantization strategy is determined as the quantization strategy for the model to be quantized, eliminating the need for quantization search in the entire quantization process, thus solving the problem of low quantization efficiency caused by the long quantization search time. If the evaluation is unsatisfactory, quantization backtracking is performed based on the first quantization strategy, improving the accuracy of some objects to be quantized in the first quantization strategy until a satisfactory second quantization strategy is obtained. Compared to the traditional method of quantization search based on large-scale parameters, this method of determining the first quantization strategy first and then backtracking reduces the search space, thus solving the problem of low model quantization efficiency caused by an excessively large search space and long search time when searching for quantization strategies.

[0099] At the same time, evaluating accuracy by using quantization strategies as units can avoid the problem of the model's accuracy exceeding the controllable range due to the accumulation of accuracy when quantizing multiple objects to be quantized. This allows the entire model to quantize objects to be quantized in multiple computation stages simultaneously, maximizing the quantization degree of the model and developing the potential of the overall quantization of the model.

[0100] The above text combined Figure 1 and Figure 2 The model quantization system provided in this application has been explained and described below, in conjunction with... Figures 3-4 This application provides an explanation of the method for determining a quantification strategy, which can be applied to... Figure 1 and Figure 2 The model quantization system 200 shown.

[0101] Figure 3 This application provides a method for determining a quantification strategy, such as... Figure 3 As shown, the method may include the following steps:

[0102] S310: Client 100 sends a model quantization request to model quantization system 200.

[0103] In specific implementation, the model quantization request may include information about the model to be quantized. This information is used by the model quantization system 200 to obtain the model data of the model to be quantized. This information may include the model file and configuration file of the model to be quantized, or the reading method of the model file and configuration file, or the identifier of the model file and configuration file, etc. It may also include other information that can enable the model quantization system 200 to obtain the model to be quantized. These will not be listed here.

[0104] Optionally, the model quantization request may also include more content, such as quantization configuration information, including calibration dataset, tolerance for accuracy loss, etc.; operating environment information, including relevant information about the hardware device on which the inference system to be deployed after model quantization is located; user-defined options, including parameter information required for the quantization report, the method of sensitivity calculation, etc., and may include even more content, which is not specifically limited in this application.

[0105] S320: The model quantization system 200 acquires the object to be quantized from the model to be quantized, as well as the data type that the object to be quantized can be quantized.

[0106] In specific implementation, the model quantization system 200 can obtain model data of the model to be quantized according to the model quantization request, and determine at least one object to be quantized that the model to be quantized supports, as well as at least one data type that each object to be quantized can be quantized. The data types that each object to be quantized can be quantized can be the same or different. For example, the model quantization system 200 can determine, according to the model data, that the objects to be quantized that the model to be quantized supports include objects A to C, wherein the data types that object A can be quantized include types 1 to 3, the data types that object B can be quantized include types 1 to 5, and the data types that object C can be quantized include types 1 to 3. The above examples are for illustration only and are not specifically limited in this application.

[0107] Optionally, the object to be quantized refers to an object in the model that can be quantized. Here, the object to be quantized is a category, and each object to be quantized may include one or more quantization parameters or quantization operators. For example, the object to be quantized may include linear transformation object to be quantized, KV cache object to be quantized, attention computation object to be quantized, and ensemble communication object to be quantized. The linear transformation object to be quantized includes operators that perform linear transformations on the data, as well as activation value parameters and weight parameters participating in the linear transformation. The KV cache object to be quantized includes the K vector and V vector in the KV cache. The attention computation object to be quantized includes operators that perform matrix multiplication calculations in the attention layer. The ensemble communication object to be quantized includes communication data during ensemble communication. Ensemble communication includes reduce scatter, all reduce, all gather, etc., and this application does not make specific limitations.

[0108] It should be understood that different types of models can quantize different objects. As mentioned above, large language models have multiple computational stages, and each stage can have one or more objects to be quantized. The following section will discuss this further. Figure 4 Taking a large language model as an example, this paper introduces the objects to be quantized in a large language model.

[0109] Figure 4 This is an example diagram of the object to be quantized in a method for determining a quantization strategy provided in this application, wherein... Figure 4 Taking a large language model with a transformer architecture as an example, this paper describes the objects that may exist in a network layer of this type of model that need to be quantized.

[0110] like Figure 4As shown, the input data is first normalized through a normalization layer to obtain a normalized result, reducing the differences in input distribution across different layers and contributing to model stability and training efficiency. Then, it enters the attention mechanism layer. The normalization result is transformed into three vectors—K vector, V vector, and Q vector—through three parallel linear transformations (matrix multiplication of the normalized result with the three weight matrices obtained during training). The K and V vectors are then cached for later reuse, reducing computational load. Next, the Q, K, and V vectors are used for attention calculation to obtain the attention result. This attention result can be input into the fully connected layer for a linear transformation (matrix multiplication of the attention result with the weight matrix and addition with the bias vector) to generate the fully connected layer output. Finally, the output is processed via ensemble communication. Algorithms, such as the allreduce communication algorithm, synchronize the output results of fully connected layers on other nodes to ensure that each node receives consistent model parameters, thus obtaining ensemble communication results. Based on these ensemble communication results, further normalization is performed to reduce the differences in input distribution across different layers, improving model stability and training efficiency. Then, a linear transformation maps the input dimension to a higher dimension, also known as up-projection computation. Next, a non-linear activation function, such as the GELU function, is used to process the output of the previous step. Then, a linear transformation maps the output data of the previous layer from a higher dimension back to the original or a lower dimension, also known as down-projection computation. Finally, ensemble communication, such as allreduce communication, synchronizes the data across all nodes to obtain the final output result.

[0111] The above describes in detail Figure 4 The steps and procedures outlined below, with a description of the quantifiable objects that can be quantified. Figure 4 The objects to be quantized were marked with dark shading.

[0112] The object to be quantized under a linear transformation, including the operators and parameters associated with the linear transformation, is... Figure 4 In this process, there are four linear transformation steps, corresponding to four objects to be quantized: generating Q, K, and V vectors through linear transformation; performing fully connected layer calculations through linear transformation; performing up calculations through linear transformation; and performing down calculations through linear transformation. These four calculations can be matrix multiplication (matmul). When quantizing this type of object, the data type of some or all of the parameters involved in the matrix multiplication calculation can be changed to reduce data precision. It should be understood that there can be one or more objects of the same type to be quantized, for example... Figure 4 In the example, there are four objects to be quantized in the linear transformation.

[0113] The KVcache contains objects to be quantized, including the K and V vectors in the KV cache. Figure 4 There is one KVcache object to be quantized. When quantizing this object, the data types of the stored K and V vectors can be changed, reducing data precision.

[0114] The objects to be quantized in ensemble communication include the communication data during ensemble communication. Figure 4 There are two sets of communication objects to be quantized. When quantizing these objects, the data type of the communication data can be changed, reducing the data precision.

[0115] Attention computation involves quantizing the object to be quantized, including the operators and parameters related to the attention computation process. Specifically, it includes the operators and parameters involved in the attention computation of the Q-vector, K-vector, and V-vector. Figure 4 There is one object to be quantized for attention calculation. Although Figure 4 There may only be one instance of matrix multiplication, but in actual attention calculations, there can be multiple instances, depending on the attention calculation method used. For example, in the Flash Attention algorithm, one matrix multiplication is the matrix multiplication of the Q vector and the K vector, used to calculate the attention score; another matrix multiplication is the matrix multiplication of the attention score and V. When quantizing this type of object, the data type of some or all of the parameters involved in the matrix multiplication can be changed, reducing data precision.

[0116] therefore, Figure 4 The scenario shown in the example includes 8 objects to be quantized, 1 KV cache object to be quantized, 1 attention computation object to be quantized, 2 set communication objects to be quantized, and 4 linear transformation objects to be quantized. It should be understood that... Figure 4 This is an example of the object to be quantized provided in this application. Other models may have more or fewer objects to be quantized, and this application does not impose any specific limitations. Furthermore, Figure 4 This is an example of an object to be quantized contained in a network layer. A model to be quantized may include M objects such as... Figure 4 The network layer shown can include at least 8M objects to be quantized in the model to be quantized.

[0117] Typically, the computation, communication, and storage steps of a model can be quantized, and there are corresponding objects to be quantized. The method for determining the quantization strategy provided in this application can quantize multiple objects to be quantized in the model simultaneously, greatly improving the quantization degree of the model. Through a series of algorithms, it avoids the accuracy loss caused by the simultaneous quantization of multiple objects to be quantized, so that the model is quantized to the maximum extent while ensuring that the model accuracy loss is within a controllable range, thus meeting the user's needs.

[0118] Optionally, data type refers to the numerical format of data in the model. Data types include floating-point types (float) and integer types (linteger). Floating-point data types include float16 (fp16), float32 (fp32), etc., where fp16 represents a 16-bit floating-point format, consisting of 1 sign bit, 5 exponent bits, and 10 mantissa bits; fp32 represents a 32-bit floating-point format, consisting of 1 sign bit, 8 exponent bits, and 23 mantissa bits. The larger the number, the greater the numerical precision, and the greater the corresponding storage and computing resource requirements. Integer data types include int8, int4, etc., where int8 represents an 8-bit integer format, representing a range from -128 to 127, or from 0 to 255 (unsigned); int4 represents a 4-bit integer format, representing a range from -8 to 7, or from 0 to 15 (unsigned). Understandably, floating-point data types have higher precision and therefore require more storage and computing resources, while integer data types have lower precision and therefore require less storage and computing resources.

[0119] For example, for linear transformation objects to be quantized, their corresponding data types may include w16a16, w8a16, w8a8, and w4a16, where w refers to the weight, a refers to the activation value, and w8a8 refers to quantizing the weight and activation value with 8 bits during matrix multiplication, making the data type of the activation value int8 and the data type of the weight int8. Similarly, w8a16 refers to quantizing the weight with 8 bits, making the data type of the weight int8 and the data type of the activation value fp16, and so on. For KV cache objects to be quantized, their corresponding data types may include 16-bit, 8-bit, and 4-bit; for set communication objects to be quantized, their corresponding data types may include 16-bit and 8-bit; for attention computation objects to be quantized, their corresponding data types may include 16-bit and 8-bit. The above examples are for illustrative purposes only and are not intended to impose specific limitations in this application.

[0120] S330: The Model Quantization System 200 combines multiple quantization strategies based on the object to be quantized and the data type.

[0121] In specific implementation, each quantization strategy includes a quantization data type corresponding to each object to be quantized. For any two quantization strategies, at least one object to be quantized has a different quantization data type in each strategy. That is, the quantization data type corresponding to each object to be quantized can be freely combined to create multiple different quantization strategies. For example, if the objects to be quantized include object 1 and object 2, where object 1 can be quantized using data types A and B, and object 2 can be quantized using data types A and B, then four quantization strategies can be obtained through free combination: in quantization strategy 1, the quantization data types of object 1 and object 2 are A and A respectively; in quantization strategy 2, the quantization data types of object 1 and object 2 are A and B respectively; in quantization strategy 3, the quantization data types of object 1 and object 2 are B and A respectively; and in quantization strategy 4, the quantization data types of object 1 and object 2 are B and B respectively. The above examples are for illustrative purposes only and are not intended to limit the specific implementation.

[0122] Furthermore, in each quantization strategy, the same quantization data type corresponds to the same object to be quantized in each network layer, so as to... Figure 4 For example, if a network layer contains 8 objects to be quantized, then the quantization data type of the objects to be quantized in the KV cache of each network layer is the same, the quantization data type of the objects to be quantized in the attention calculation of each network layer is the same, and so on.

[0123] For example, Figure 4 In the example shown, a network layer includes 8 objects to be quantized. The linear transformation objects can be quantized using data types including w16a16, w8a16, w8a8, and w4a16. The ensemble communication objects can be quantized using 16-bit and 8-bit data types. The KV cache objects can be quantized using 16-bit, 8-bit, and 4-bit data types. The attention computation objects can be quantized using 16-bit and 8-bit data types. (Reference...) Figure 4 As shown in the examples, there are 4 objects to be quantized for linear transformation, each with 4 quantizable data types; 2 objects to be quantized for set communication, each with 2 quantizable data types; 1 object to be quantized for KVcache, with 3 quantizable data types; and 1 object to be quantized for attention computation, with 2 quantizable data types, for a total of 8 objects to be quantized. These can be freely combined to obtain 6144 quantization strategies. The above examples are for illustrative purposes only and are not intended to limit the scope of this application.

[0124] It should be understood that each network layer includes 8 objects to be quantized. If each object to be quantized in each network layer is freely combined according to different data types, the number of quantization strategies may be very large. Performing sensitivity calculations for each quantization strategy may generate computational pressure. Therefore, combining multiple quantization strategies according to the same data type of the objects to be quantized can effectively reduce computational pressure.

[0125] S340: The model quantization system 200 determines the sensitivity of each quantization strategy and obtains the first quantization strategy based on the sensitivity.

[0126] In this embodiment, the model quantization system 200 can calculate the sensitivity of each quantization strategy. This sensitivity indicates the degree of influence of the quantization strategy on the model accuracy. The model quantization system 200 can select the quantization strategy with the least impact on model accuracy from multiple quantization strategies based on the sensitivity as the first quantization strategy. Specifically, the model quantization system 200 can first calculate the sensitivity of each object to be quantized in the quantization strategy, and then determine the sensitivity of the entire quantization strategy based on the sensitivity of each object. For example, it can sum the sensitivities of each object to obtain the sensitivity of the entire quantization strategy, or perform a weighted summation. This application does not impose specific limitations. Specifically, if the sensitivity is higher, it indicates a greater impact of the quantization strategy on model accuracy. In this case, the sensitivities of multiple quantization strategies can be sorted, and the quantization strategy with the lowest sensitivity can be selected as the first quantization strategy.

[0127] Furthermore, the sensitivity can be obtained based on one or more sensitive parameters related to the model accuracy. When calculating the sensitivity of each object to be quantized, one or more sensitive parameters can be calculated first, and then the sensitivity can be determined. For example, the sensitivity can be obtained based on the product of one or more sensitive parameters, or based on the weighted sum of one or more sensitive parameters. This application does not make any specific limitations.

[0128] Optionally, the sensitive parameter may include the average trace of the Hessian matrix of the weights or activations in the object to be quantized. It should be understood that the smaller the average trace of the Hessian matrix of the weights, the greater the accuracy loss to the entire model due to the quantization of those weights. Therefore, using the average trace as a sensitive parameter can effectively indicate the impact of the quantization strategy on model accuracy. Similar to weights, the average trace of activations can also effectively indicate their impact on model accuracy after quantization. Likewise, the weights and activations in the objects to be quantized in ensemble communication, attention computation, and KV cache can also be evaluated using the average trace to assess the impact of their quantization strategies on model accuracy.

[0129] In practical implementation, when calculating the average trace of the Hessian matrix of the weights of the linear transformation quantization operator, the calculation of the average trace of the Hessian matrix is ​​relatively time-consuming, and the trace(H) of a large matrix (identifying the trace of the Hessian matrix) is difficult to calculate. However, the bilinear calculation form z T Hz (where z is a random vector) is relatively easy to compute. Therefore, the Hutchinson algorithm can be used, based on z. T The average value E(z) in Hz form T The average trace (H) of the weighted Hessian matrix can be quickly estimated using the formula (1).

[0130]

[0131] Select m random vectors z that conform to a Gaussian distribution. i H is the weighted Hessian matrix. The average trace of the Hessian matrix can be quickly calculated using formula (1), which is based on the average trace of the weighted Hessian matrix.

[0132] In the specific implementation, when calculating the average trace of the Hessian matrix of the activation values ​​of the linear transformation quantization operator, since each input to the model is independent, the Hessian matrix of the intermediate layer activations is a block diagonal matrix, with each block corresponding to an input x. i Or another input x i+1 The Hessian matrix of the activation layer is the average trace of the Hessian matrix. This can be obtained by averaging the traces of the Hessian matrices corresponding to typical model inputs. Based on the Hutchinson algorithm, by selecting a certain number of inputs, the average trace of the activation Hessian matrix will gradually converge. Therefore, the average trace of the activation Hessian matrix can be quickly and accurately estimated based on formula (2):

[0133]

[0134] Where N is the number of model inputs, x i z is the i-th input to the model. i H is an element of a random vector z that follows a Gaussian distribution. aj(xi) The model input x i The Hessian matrix of the activation of the j-th layer.

[0135] The above text describes in detail the average trace of the Hessian matrices of the activation values ​​and weights of the linear transformation quantization operator. Since the input data of the objects to be quantized in KVCache, attention calculation, and set communication are also dynamically changing, they belong to the activation values. Referring to the relevant description in formula (2) above, the average trace of the Hessian matrices of the activation values ​​of the objects to be quantized in KVCache, attention calculation, and set communication can be obtained. Specifically, for the objects to be quantized in KVCache, the sensitivity can be obtained based on the average trace of the Hessian matrices of the K and V vectors stored in the KV cache. For the objects to be quantized in attention calculation, the sensitivity can be obtained based on the average trace of the Hessian matrices of the Q, K, and V vectors involved in attention calculation. For the objects to be quantized in set communication, the sensitivity can be obtained based on the average trace of the Hessian matrices of the data to be communicated participating in set communication. Examples are not provided here.

[0136] It should be understood that the above example provides a way to calculate the average trace of the Hessian matrix. In specific implementations, the average trace can also be calculated in other ways, which will not be illustrated here.

[0137] Optionally, the sensitive parameters may also include the loss value, which is the loss function value calculated based on the output results of the model after quantization and before quantization. For example, after converting 32-bit floating-point numbers to 16-bit floating-point numbers, the mean squared error (MSE) values ​​under the two precisions are calculated. The above examples are for illustration only and are not specifically limited in this application.

[0138] It should be understood that the above examples provide some possible sensitive parameters. In specific implementations, other sensitive parameters can also be combined to determine the sensitivity, and this application does not impose specific limitations. After the sensitive parameters of each object to be quantized are determined, its corresponding sensitivity can be obtained through certain calculation methods, such as multiplying the average trace and the loss value to obtain its corresponding sensitivity, or obtaining its corresponding sensitivity through weighted summation, and this application does not impose specific limitations.

[0139] Optionally, after obtaining the sensitivity of each object to be quantized, the sensitivity of the entire quantization strategy can be determined based on the sensitivity of each object. As mentioned above, the quantization strategy is obtained through free combination. Figure 4The large language model shown includes 8 objects to be quantized, namely 1 KV cache object, 1 attention computation object, 2 set communication objects, and 4 linear transformation objects. The data types that each object can be quantized are as described in the previous example. Therefore, there are 6144 quantization strategies that can be freely combined. The sensitivity Ω of each quantization strategy can be calculated according to the following formula (3):

[0140]

[0141] Where i refers to the i-th layer of the large language model, and L refers to the total number of layers in the large language model. It is the average trace of the Hessian matrix of the entire group of 8 operators. It is the L2 norm of the quantization perturbation of the i-th layer, which is the L2 norm of the error between the output after quantization and the output before quantization, and is the loss value mentioned above. Simply put, formula (3) multiplies the average trace of the Hessian matrix of the weights or activations with the loss value and sums the results across the entire network to obtain the sensitivity of the entire quantization strategy. Each quantization strategy is calculated in this way, and the quantization strategy with the lowest sensitivity is the optimal first quantization strategy.

[0142] It should be understood that since Ω reflects the overall sensitivity of all 8 objects to be quantized in the model, it is an assessment of the accuracy error of the entire quantization strategy. It takes into account the cumulative error of various data type combinations on the accuracy of the entire network. The first quantization strategy determined in this way takes into account the cumulative error of the quantization of the 8 objects to be quantized, and is the best optimization scheme, obtaining the most suitable quantization data type for each object to be quantized.

[0143] S350: The model quantization system 200 evaluates the first quantization strategy and obtains the quantization evaluation results.

[0144] Optionally, the model quantization system 200 can first perform outlier suppression based on a first quantization strategy to obtain a suppressed first quantization strategy, and then perform quantization evaluation on the suppressed first quantization strategy to obtain quantization evaluation results. Outlier suppression refers to identifying and adjusting extreme outliers in the data to reduce the amplitude impact of these outliers on the quantization process. Outliers are values ​​that are significantly different from the overall data distribution. Outliers may be caused by low-probability events and are not representative, but they can still affect the overall data distribution.

[0145] In its implementation, the model quantization system 200 can obtain the data distribution of activation values ​​of the object to be quantized based on model forward inference. Using an outlier identification algorithm, it identifies outliers, and then uses an outlier suppression algorithm to adjust these outliers, making the data distribution smoother, thus obtaining a suppressed first quantization strategy. The outlier identification algorithm can include the interquartile range method (IQR) and the standard score method (Z-score) algorithm, detecting discrete values ​​that deviate from the main distribution. These discrete values ​​may adversely affect the accuracy of quantization. The outlier suppression algorithm can employ mathematically equivalent transformation methods, such as staged or tail-reducing processing of extreme activation values, to make the data distribution smoother and reduce quantization errors.

[0146] Optionally, when evaluating the first quantization strategy, quantization calibration can be performed first to obtain the quantization parameters for each object to be quantized. Then, these quantization parameters are used to quantize the object to be quantized, resulting in a quantized model to be evaluated. Finally, the model to be evaluated is run on a test set to obtain the quantization evaluation results. The quantization evaluation results may include metrics such as accuracy, inference speed, resource consumption, model file size, robustness, and stability, which are not specifically limited in this application.

[0147] In specific implementation, quantization calibration is used to determine quantization parameters to reduce accuracy loss during quantization. Quantization parameters may include a scaling factor and an offset. The scaling factor adjusts the scale of the data during quantization, allowing the data to be mapped from a high-precision format to a low-precision format range. For example, if the first range of the original data is from -1.5 to 1.5, and the second range of the quantized data is from 0 to 255, the scaling factor is used to map the first range to the second range. The offset is used to adjust the center point of the data to a suitable position to ensure that the quantized data reflects the statistical characteristics of the original data as much as possible. The algorithms used to determine the quantization parameters may include min-max normalization, Klöbek-Leibler divergence, etc., and may also include other methods for effectively mapping data to low-precision numerical orientations; this application does not specifically limit these methods.

[0148] It should be understood that although the quantized model uses a low-precision quantization data type, determining appropriate quantization parameters through quantization calibration can ensure that the model output is not significantly affected. Furthermore, quantization calibration of multiple objects to be quantized can be performed simultaneously, sequentially, or partially simultaneously and partially sequentially, depending on actual business needs; this application does not impose specific limitations.

[0149] S360: The model quantization system 200 determines whether the quantization evaluation result is qualified. If yes, that is, if the quantization evaluation result is qualified, proceed to S380; if no, that is, if the quantization evaluation result is unqualified, proceed to S370.

[0150] Optionally, based on the quantitative evaluation results, the indicators before and after quantization can be compared to obtain indicator loss values. The pass / fail status of the quantitative evaluation results can be determined based on the difference between the indicator loss values ​​and a threshold. For example, the accuracy loss value can be obtained by comparing the model accuracy before quantization (e.g., classification accuracy, regression accuracy) with the model accuracy after quantization; the stability loss value can be obtained by comparing the model stability score before quantization (e.g., noise resistance, generalization ability) with the model stability score after quantization. The pass / fail status of the quantitative evaluation results can be determined based on the loss values ​​of multiple indicators. Furthermore, the quantitative evaluation results can also be scored based on a pre-defined evaluation system, and the pass / fail status of the quantitative evaluation results can be determined based on the score. This application does not impose specific limitations on this approach.

[0151] S370: The model quantization system 200 performs quantization back-off on the first quantization strategy to obtain the second quantization strategy.

[0152] In practical implementation, if the quantization evaluation result shows that the first quantization strategy is unqualified, quantization backtracking can be performed on the first quantization strategy. This involves backtracking the quantization data type of some objects to be quantized in the first quantization strategy to a higher precision, thus obtaining a second quantization strategy that passes the evaluation. Referring to the foregoing, in the first quantization strategy, the quantization data type of the same objects to be quantized is the same in each network layer. Therefore, S370 can perform quantization backtracking according to network layers, sequentially determining the quantization data type of each object to be quantized in each network layer until a second quantization strategy that passes the evaluation is obtained. Thus, in the final second quantization strategy, the quantization data type of the same object to be quantized may differ in different network layers. For example, the quantization data type of the objects to be quantized in the KV cache of the first network layer may be 16 bits, while the quantization data type of the objects to be quantized in the KV cache of the second network layer may be 8 bits. The above examples are for illustrative purposes only and are not intended to limit the scope of the application.

[0153] It should be understood that this application first selects a relatively superior first quantization strategy through coarse-grained screening in steps S310 to S360. In this first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. Then, through quantization back-off according to each network layer in S370, a more fine-grained second quantization strategy can be obtained. Compared with the traditional method of sequentially searching for the corresponding quantization data type of a large number of parameters, this method of coarse-grained screening followed by fine-grained back-off can reduce the search space, thereby improving the efficiency of quantization of large language models.

[0154] Optionally, to further reduce the search space, the model quantization system 200 can pre-set M backoff schemes. Each backoff scheme includes the objects to be quantized and the data types that the objects to be quantized can be quantized. The objects to be quantized may be different in different backoff schemes, and the data types that the objects to be quantized can be quantized may also be different. For example, backoff scheme 1 includes objects to be quantized 1 and objects to be quantized 2, backoff scheme 2 includes objects to be quantized 1, objects to be quantized 2, and objects to be quantized 3, backoff scheme 3 includes objects to be quantized 1 and objects to be quantized 3, and so on. Then, according to the search order, the optimal quantization strategy under each backoff scheme is determined sequentially, and the optimal quantization strategy is evaluated. If the evaluation result is satisfactory, the quantization strategy is determined as the second quantization strategy. If it is unsatisfactory, the optimal quantization strategy under the next backoff scheme is determined, and so on, until a satisfactory second quantization strategy is obtained. In this way, only a maximum of M evaluations are needed to obtain the second quantization strategy, which can improve the efficiency of quantization backoff.

[0155] In practical implementation, when determining the optimal quantization strategy for each fallback scheme, the sensitivity of the object to be quantized can be calculated in the same way as the method described above for determining the first quantization strategy. The quantization strategies are then ranked based on these sensitivity scores to obtain the optimal quantization strategy for that fallback scheme. It should be noted that when determining the optimal quantization strategy for each fallback scheme, the quantization strategy for each network layer is also determined based on the sensitivity ranking method, thus obtaining the optimal quantization strategy for that fallback scheme.

[0156] For example, suppose the optimal quantization strategy in rollback scheme 1 is determined. Rollback scheme 1 includes the KV cache object to be quantized, the linear transformation object to be quantized, and the attention calculation object to be quantized. At this time, the number of objects to be quantized is 6. The data types of the 8 objects to be quantized in the first network layer can be freely combined to obtain m1 quantization strategies. Then, the sensitivity of the m1 quantization strategies in this network layer is calculated according to the above formulas (1) to (3). Then, the optimal quantization strategy is determined. Then, the optimal quantization strategy of the second network layer is calculated. After all N network layers have obtained their corresponding optimal quantization strategies, the optimal quantization strategy of rollback scheme 1 is obtained. The quantization strategy includes the quantization data type of each object to be quantized in each network layer. The quantization data type of the same object to be quantized in different network layers may be different. At this time, the optimal quantization strategy is quantized and evaluated. If it is qualified, it can be determined as the second quantization strategy. If it is not qualified, the optimal quantization strategy in rollback scheme 2 is determined. The determination process can be referred to the above description, which will not be elaborated here.

[0157] Furthermore, the search order of the rollback scheme can be determined according to the number of objects to be quantized in the rollback scheme. The more objects to be quantized in the rollback scheme, the higher the priority of the search order. In other words, during the search, the number of objects to be quantized participating in the search is gradually reduced, and the number of objects to be quantized for rollback is gradually increased. For example, if there are 5 objects to be quantized, first search according to 4 objects to be quantized, which means rollback 1 object to be quantized. Then search according to 3 objects to be quantized, which means rollback 2 objects to be quantized. Then search according to 2 objects to be quantized, which means rollback 3 objects to be quantized, and so on. By gradually reducing the number of objects to be quantized in this way, large-scale rollback can be avoided and rollback efficiency can be improved.

[0158] For example, with Figure 4 Taking a scenario as an example, suppose the first quantization strategy includes 8 objects to be quantized. Table 1 is an example table of rollback schemes. As shown in Table 1, schemes 5 and 4 include 3 objects to be quantized, while schemes 3, 2, and 1 include 2 objects to be quantized. Therefore, during quantization rollback, the optimal quantization strategy (also called the third quantization strategy) under each rollback scheme can be determined sequentially according to the order of schemes 5, 4, 3, 2, and 1. Then, the quantization strategies are evaluated. If they are not satisfactory, the optimal quantization strategy under the next rollback scheme is determined, until a satisfactory second quantization strategy is obtained. In this way, a maximum of 5 evaluations are needed to obtain the second quantization strategy, reducing the number of quantization evaluations and improving quantization efficiency.

[0159] Table 1. Example of rollback plan

[0160]

[0161]

[0162] It should be noted that if no quantitative strategy achieves a satisfactory evaluation result under the last rollback plan, a notification message can be sent to the user indicating that manual quantitative rollback is required, prompting the user to perform manual rollback. The user can then manually set a second quantitative strategy based on their experience.

[0163] In practical implementation, when prompting the user to manually roll back, the message can include the optimal quantization strategy and quantization evaluation results for each rollback option. Although these quantization strategy evaluation results are unsatisfactory, they can still be used as a reference for the user, facilitating a faster acquisition of a quantization strategy for manual rollback. Furthermore, the quantization evaluation results of the optimal quantization strategy for each rollback option can be analyzed, and the analysis results can be displayed to the user, such as showing the mean squared error of each quantization strategy. This helps the user determine which objects to be quantized and which network layers have larger errors, allowing the user to modify the quantization data type of the objects to be quantized and obtain a second quantization strategy.

[0164] It should be understood that during quantization backoff, searching for quantization strategies in a more granular manner and completing the refined search in a certain search order can not only obtain more refined and accurate quantization strategies, but also improve search efficiency. Compared with the approach of searching directly in the massive search space of multiple network layers, multiple objects to be quantized, and multiple data types, the technical solution of this application can obtain the optimal quantization strategy more efficiently and with higher quality, thereby improving the quantization efficiency of the model.

[0165] S380: The model quantization system 200 quantizes the model to be quantized and obtains the model quantization result.

[0166] In practice, if the evaluation result of the first quantization strategy is qualified, the first quantization strategy can be used to quantize the model to be quantized and obtain the model quantization result without executing S360. If the evaluation result of the first quantization strategy is unqualified, the second quantization strategy determined by S360 can be used to quantize and obtain the model quantization result.

[0167] In practice, the model quantization result includes quantized model data, quantization strategy, quantization weight, quantization parameters, etc. The specific details can be determined according to actual business needs, and this application does not impose any specific limitations.

[0168] S390A: The model quantization system 200 sends a quantization report to the client 100.

[0169] In practice, a quantification report can be generated based on the model quantification results. The quantification report may include the aforementioned model quantification results, relevant information before model quantification, and analytical data for analyzing the model quantification results. For example, some indicator data in the quantification evaluation results may be processed in the form of charts, some possible risks of the quantified model may be analyzed, and corresponding risk control measures may be proposed. The specific content of the quantification report may also be determined according to actual business needs. This application does not impose specific limitations.

[0170] S390B: The model quantization system 200 sends the model quantization results to the inference system 300.

[0171] Optionally, the model quantization system 200 can execute S390A and S390B simultaneously. Alternatively, after S390A is completed, the user determines to use the second quantization strategy and sends a confirmation message to the model quantization system 200 through the client 100, after which the model quantization system 200 executes S390B. This application does not impose any specific limitations.

[0172] It should be understood that after receiving the model quantization result, the inference system 300 can use the quantization weights and parameters to load the quantization weights and parameters into the inference engine for inference deployment. Quantization reduces the memory space required by the model, making it easier to deploy on memory-constrained devices such as mobile devices, embedded systems, and edge devices. It also accelerates the inference process, which is particularly important for applications requiring real-time processing, such as real-time translation and dialogue systems. Furthermore, it reduces the power consumption required for model operation, a significant advantage for battery-powered mobile devices.

[0173] In summary, this application provides a method for determining a quantization strategy. This method involves freely combining the objects to be quantized supported by the model to be quantized and the data types that can be quantized to obtain multiple different quantization strategies. Then, a sensitivity algorithm is used to determine the sensitivity of different quantization strategies, which indicates the impact of the quantization strategy on the accuracy of the model to be quantized. A first quantization strategy is determined based on the sensitivity of each strategy, and then the first quantization strategy is evaluated. If the evaluation is satisfactory, the first quantization strategy is determined as the quantization strategy for the model to be quantized, eliminating the need for quantization search in the entire quantization process, thus solving the problem of low quantization efficiency caused by the long quantization search time. If the evaluation is unsatisfactory, quantization backtracking is performed based on the first quantization strategy, improving the accuracy of some objects to be quantized in the first quantization strategy until a satisfactory second quantization strategy is obtained. Compared to the traditional method of quantization search based on large-scale parameters, determining the first quantization strategy first and then backtracking reduces the search space, thus solving the problem of excessive search space and long search time during model quantization, which leads to low model quantization efficiency.

[0174] At the same time, evaluating accuracy by using quantization strategies as units can avoid the problem of the model's accuracy exceeding the controllable range due to the accumulation of accuracy when quantizing multiple objects to be quantized. This allows the entire model to quantize objects to be quantized in multiple computation stages simultaneously, maximizing the quantization degree of the model and developing the potential of the overall quantization of the model.

[0175] The architecture of the model quantization system and the method for determining the quantization strategy provided in this application have been described in detail above. The following section will combine... Figure 5 The internal structure of the model quantization system provided in this application is explained.

[0176] Figure 5 This is a schematic diagram of the model quantization system provided in this application. The model quantization system 200 can be the model quantization system described above. For example... Figure 5As shown, the model quantization system 200 includes an acquisition unit 510, a combination unit 520, a determination unit 530, an evaluation unit 540, a backtracking unit 550, a quantization unit 560, and a prompting unit 570. It should be understood that... Figure 5 This is an exemplary partitioning method. The model quantization system 200 can be divided into more or fewer unit modules, and this application does not make any specific limitations.

[0177] The acquisition unit 510, combination unit 520, determination unit 530, evaluation unit 540, backoff unit 550, quantization unit 560, and prompting unit 570 can all be implemented in software or hardware. For example, the implementation of the acquisition unit 510 will be described below. Similarly, the implementation of the combination unit 520, determination unit 530, evaluation unit 540, backoff unit 550, quantization unit 560, and prompting unit 570 can refer to the implementation of the acquisition unit 510.

[0178] As an example of a software functional unit, the acquisition unit 510 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition unit 510 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0179] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0180] As an example of a hardware functional unit, the acquisition unit 510 may include at least one computing device, such as a server. Alternatively, the acquisition unit 510 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-a-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0181] The multiple computing devices included in the acquisition unit 510 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition unit 510 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition unit 510 can be distributed in the same VPC or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0182] It should be noted that, in other embodiments, the steps implemented by the acquisition unit 510, combination unit 520, determination unit 530, evaluation unit 540, rollback unit 550, quantization unit 560, and prompting unit 570 can be specified as needed. The acquisition unit 510, combination unit 520, determination unit 530, evaluation unit 540, rollback unit 550, quantization unit 560, and prompting unit 570 respectively implement different steps in the method of determining the quantization strategy to realize all the functions of the model quantization system 200.

[0183] The functions of the acquisition unit 510, combination unit 520, determination unit 530, evaluation unit 540, rollback unit 550, quantization unit 560 and prompting unit 570 are explained below.

[0184] Acquisition unit 510 is used to acquire one or more objects to be quantized from the model to be quantized, and at least one data type that each object to be quantized can be quantized, specifically for implementing... Figure 3 S310, S320 and their optional steps in the embodiments.

[0185] Combination unit 520 is used to determine multiple quantization strategies based on at least one data type that each object to be quantized can quantize. Each quantization strategy includes a quantization data type for each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can quantize. Specifically, it is used to implement... Figure 3 S330 and its optional steps in the embodiments.

[0186] The determining unit 530 is used to determine the sensitivity of each quantization strategy, wherein the sensitivity indicates the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy. Based on the sensitivity of each quantization strategy, a first quantization strategy is obtained from multiple quantization strategies, wherein the first quantization strategy is used to quantize one or more objects to be quantized, specifically for implementing... Figure 3 S340 and its optional steps in the embodiments.

[0187] In one possible implementation, the determining unit 530 is used to determine the sensitivity of each object to be quantized in each quantization strategy, and to obtain the sensitivity of each quantization strategy based on the sensitivity of each object to be quantized. The sensitivity of each object to be quantized is obtained based on one or more parameters, including the average trace of the Hessian matrix of the object to be quantized and the loss values ​​of the object before and after quantization. Specifically, it is used to implement... Figure 3 S340 and its optional steps in the embodiments.

[0188] In one possible implementation, the evaluation unit 540 is used to perform quantitative evaluation on the first quantization strategy and obtain the quantitative evaluation result, specifically for implementing... Figure 3 The embodiments include steps S350-S360 and their optional steps. The rollback unit 550 is used to roll back the first quantization strategy when the quantization evaluation result is unqualified. This involves modifying the quantization data type of the object to be quantized in the first quantization strategy to obtain a second quantization strategy. The second quantization strategy is used to quantize one or more objects to be quantized. Specifically, it is used to implement... Figure 3 S380 and its optional steps in the embodiments.

[0189] In one possible implementation, the quantization unit 560 is used to quantize the model to be quantized using a first quantization strategy or a second quantization strategy, obtain the quantized model, and send the quantized model to the inference system. Specifically, it is used to implement... Figure 3 S380, S390A, S390B and their optional steps in the embodiments.

[0190] In one possible implementation, the model to be quantized includes multiple network layers. In the first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. In the second quantization strategy, the quantization data type of the same object to be quantized is the same or different in each network layer. The number of objects to be quantized in the second quantization strategy is less than the number of objects to be quantized in the first quantization strategy.

[0191] In one possible implementation, the rollback unit 550 is used to acquire multiple rollback schemes, each rollback scheme including at least one object to be quantized, and the number of objects to be quantized in different rollback schemes is different. The rollback unit 550 is used to determine the quantization data type of each object to be quantized in the first rollback scheme among the multiple rollback schemes, and obtain the third quantization strategy corresponding to the first rollback scheme. The rollback unit 550 is used to perform quantization evaluation on the third quantization strategy to obtain the rollback evaluation result. If the rollback evaluation result is qualified, the rollback unit 550 is used to determine the third quantization strategy as the second quantization strategy. If the rollback evaluation result is unqualified, the rollback unit 550 is used to determine the quantization data type of each object to be quantized in the second rollback scheme, and obtain the fourth quantization strategy corresponding to the second rollback scheme. The number of objects to be quantized in the first rollback scheme is not less than the number of objects to be quantized in the second rollback scheme.

[0192] In one possible implementation, the backoff unit 550 is used to obtain at least one level quantization strategy for each network layer under the first backoff scheme based on the objects to be quantized included in the first backoff scheme and the data type that each object to be quantized can be quantized; the backoff unit 550 is used to determine the sensitivity of each level quantization strategy of each network layer; based on the sensitivity of each level quantization strategy, determine the quantization strategy of each network layer; and the backoff unit 550 is used to obtain a third quantization strategy corresponding to the first backoff scheme based on the quantization strategy of each network layer.

[0193] In one possible implementation, the objects to be quantized include linear transformation objects, key-value cache (KVcache) objects, set communication objects, and attention computation objects. The linear transformation objects include operators that perform linear transformations on the data, activation parameters, and weight parameters. The KVcache objects include K and V vectors in the KVcache. The attention computation objects include operators that perform matrix multiplication in the attention layer. The set communication objects include communication data during set communication.

[0194] In one possible implementation, the prompting unit 570 is used to analyze the back-off evaluation results corresponding to each back-off scheme when all back-off evaluation results of each back-off scheme are unqualified, and obtain analysis results. The analysis results include the quantized object and network layer information whose impact on quantization error is greater than the threshold. The prompting unit 570 is used to send a prompt message for manual back-off to the client. The prompt message includes one or more of the following: the third quantization strategy corresponding to each back-off scheme, the back-off evaluation results corresponding to each third quantization strategy, and the analysis results.

[0195] In summary, the model quantization system provided in this application obtains multiple quantization strategies by freely combining the objects to be quantized and the data types that can be quantized, supported by the model to be quantized. Then, based on a sensitivity algorithm, the sensitivity of different quantization strategies is determined. This sensitivity indicates the impact of the quantization strategy on the accuracy of the model to be quantized. A first quantization strategy is determined based on the sensitivity of each quantization strategy, and then the first quantization strategy is evaluated. If the evaluation is qualified, the first quantization strategy is determined as the quantization strategy of the model to be quantized, so that the entire quantization process does not require quantization search, thus solving the problem of low quantization efficiency caused by long quantization search time in model quantization. If the evaluation is unqualified, quantization backtracking is performed based on the first quantization strategy to improve the accuracy of some objects to be quantized in the first quantization strategy until a qualified second quantization strategy is obtained. Compared with the traditional method of quantization search based on large-scale parameters, the method of determining the first quantization strategy first and then backtracking can reduce the search space, thus solving the problem of low model quantization efficiency caused by excessive search space and long search time when searching for quantization strategies during the model quantization process.

[0196] At the same time, evaluating accuracy by using quantization strategies as units can avoid the problem of the model's accuracy exceeding the controllable range due to the accumulation of accuracy when quantizing multiple objects to be quantized. This allows the entire model to quantize objects to be quantized in multiple computation stages simultaneously, maximizing the quantization degree of the model and developing the potential of the overall quantization of the model.

[0197] The foregoing has described in detail the method for determining the quantization strategy and the model quantization system provided in this application. The following section will combine... Figures 6-8 The computing device provided in this application will be explained.

[0198] Figure 6 This is a schematic diagram of the structure of a computing device provided in this application, such as... Figure 6As shown, computing device 600 includes: bus 602, processor 604, memory 606, accelerator card 607, and communication interface 608. Processor 604, memory 606, and communication interface 608 communicate via bus 602. Computing device 600 can be a server. It should be understood that this application does not limit the number of processors, memory, and accelerator cards in computing device 600. Computing device 600 can be used to deploy the aforementioned... Figures 1-5 The model quantization system in this embodiment. Memory 606 is used to store instructions, and processor 604 is used to execute the instructions stored in memory 606. Memory 606 stores program code, and processor 604 can call the program code stored in memory 606 to perform the following operations:

[0199] Obtain one or more objects to be quantized from the model to be quantized, and at least one data type that each object can be quantized from:

[0200] Based on at least one data type that each object to be quantized can be quantized, multiple quantization strategies are determined, wherein each quantization strategy includes the quantization data type of each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can be quantized.

[0201] Determine the sensitivity of each quantization strategy, where sensitivity indicates the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy;

[0202] Based on the sensitivity of each quantization strategy, a first quantization strategy is obtained from multiple quantization strategies, wherein the first quantization strategy is used to quantize one or more objects to be quantized.

[0203] It should be understood that the memory 606 stores executable program code, and the processor 604 executes the executable program code to achieve the above. Figure 1 and Figure 5 The operational steps of the method for determining a quantization strategy described in each embodiment are as follows. The program code may include one or more software units, which are... Figure 5 The embodiment includes an acquisition unit, a combination unit, a determination unit, an evaluation unit, a backoff unit, a quantization unit, and a prompting unit. The acquisition unit is used to implement... Figure 3 The S310, S320 and their optional steps in the embodiment are combined by the unit to implement Figure 3 S330 and its optional steps in the embodiment, the determining unit is used to implement Figure 3 S340 and its optional steps in the embodiment, the evaluation unit is used to implement Figure 3 S350 of the embodiment and its optional steps, the back-off unit is used to implement Figure 3 S360 of the embodiment and its optional steps, the quantization unit is used to implement Figure 3 S370 and its optional steps in the embodiment, the prompting unit is used to implement Figure 3 The steps S390A to S390B and their optional steps in the embodiment will not be described in detail here.

[0204] Bus 602 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 602 is represented by only one line, but this does not mean that there is only one bus or one type of bus. The bus 602 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608). The unified bus may also be called the Lingqu bus.

[0205] Processor 604 may include any one or more computing devices such as central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0206] Memory 606 may include volatile memory, such as random access memory (RAM). Memory 606 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 606 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0207] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 606, or two or more types of storage media can be configured to realize the function of memory 606. This application does not limit this.

[0208] The communication interface 608 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other devices or communication networks.

[0209] As one possible implementation method, Figures 1 to 5 The method shown can be implemented either by the processor in computing device 600 executing program code in memory, or by the accelerator card 607 in computing device 600. The accelerator card 607 can be implemented by computing devices such as DPU, NPU, GPU, XPU, SoC, or AI chips; this application does not specifically limit its implementation. The accelerator card 607 includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to execute the above-described... Figures 1 to 5 The operational steps of the method for determining the quantization strategy described in each embodiment are as follows.

[0210] As one possible implementation method, Figures 1 to 5 The method described above can be implemented either by the processor in computing device 600 executing program code in memory, or by using accelerator card 607, or by a combination of processor 604 and accelerator card 607 in computing device 600. Figures 1 to 5 The method for determining the quantization strategy described herein refers to the operational steps performed by the corresponding entity. Specifically, the program code stored in memory 606 includes... Figure 5The partial unit modules shown enable the processor 604 to execute part of the operation steps of the method for determining the quantization strategy by executing the program code, and the power supply circuit in the accelerator card 607 supplies power to the processing unit, enabling the processing unit to execute another part of the operation.

[0211] For example, memory 606 stores program code for acquisition unit, combination unit, and prompting unit, and the processing unit of accelerator card 607 is used to implement the functions of determination unit, evaluation unit, quantization unit, and rollback unit. The above examples are for illustration only and are not specifically limited in this application. Furthermore, the functional descriptions of the above multiple units can be found in the relevant content above, and will not be repeated here.

[0212] This application also provides a computing device cluster, such as Figure 7 As shown, Figure 7 This is an example diagram of a computing device cluster provided in this application, which may include at least one Figure 6 The computing device of the embodiment is used to deploy the model quantization system 200 described above.

[0213] As one possible implementation, the same implementation can be stored in the memory 606 of one or more computing devices 600 in the computing device cluster. Figures 1 to 5 The instructions describe the operational steps of the method for determining the quantization strategy. Alternatively, different implementations may be stored separately in the memory 606 of one or more computing devices 600. Figures 1 to 5 The instructions for the operational steps of the method for determining a quantization strategy are described. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for the operational steps required to implement the method for determining a quantization strategy.

[0214] As one possible implementation, the same implementation can be stored in the accelerator cards 607 of one or more computing devices 600 in the computing device cluster. Figures 1 to 5 The instructions describe the operational steps of the method for determining the quantization strategy. Alternatively, different implementations may be stored in one or more accelerator cards 607 of the computing device 600. Figures 1 to 5 The instructions for the operational steps of the method for determining a quantization strategy are described. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for the operational steps required to implement the method for determining a quantization strategy.

[0215] As one possible implementation, the same implementation is stored in the memory 606 of one or more computing devices 600 in the computing device cluster. Figures 1 to 5The instructions for a portion of the operational steps in the described method for determining the quantization strategy are stored in the accelerator card 607, along with the instructions for implementing another portion of the operational steps. Alternatively, different implementations may be stored in the memory 606 of one or more computing devices 600. Figures 1 to 5 The instruction for one part of the operation steps of the described data method is stored in the accelerator card 607, and different instructions for implementing another part of the operation steps are stored in the accelerator card 607. In other words, a combination of one or more computing devices 600 can jointly execute the instruction for the operation steps required to implement the method for determining the quantization strategy.

[0216] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, Figure 8 This is a schematic diagram of another computing device cluster structure provided in this application. Two computing devices, 600A and 600B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 606 in computing device 600B stores instructions for acquisition, combination, and prompting functions. Meanwhile, the accelerator card 607 in computing device 600A is used to implement the functions of a determination unit, evaluation unit, backoff unit, and quantization unit.

[0217] Figure 8 The connection method between the computing device clusters shown can be based on the fact that data query systems are usually distributed systems. In order to reduce the processing pressure of each device, the unit modules are distributed on different computing devices.

[0218] It should be understood that Figure 8 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0219] This application embodiment also provides a chip, which includes a processing unit and a power supply circuit. The power supply circuit supplies power to the processing unit, enabling the processing unit to perform... Figures 1 to 5 The operational steps of determining the quantization strategy in the embodiments are omitted here for the sake of brevity.

[0220] This application also provides a chip system comprising multiple chips as described above, which is used to deploy the model quantization system 200 described above.

[0221] This application also provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the method for determining a quantization strategy provided in this application.

[0222] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a reasoning method based on a large language model, or instruct the computing device to execute a reasoning method.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for determining a quantification strategy, characterized in that, The method is applied to a model quantization system, and the method includes: Obtain one or more objects to be quantized from the model to be quantized, and at least one data type that each object can be quantized; Based on at least one data type that each object to be quantized can be quantized, multiple quantization strategies are determined, wherein each quantization strategy includes a quantization data type for each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can be quantized. Determine the sensitivity of each quantization strategy, wherein the sensitivity is used to indicate the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy; Based on the sensitivity of each quantization strategy, a first quantization strategy is obtained from the plurality of quantization strategies, wherein the first quantization strategy is used to quantize the one or more objects to be quantized.

2. The method according to claim 1, characterized in that, Determining the sensitivity of each quantization strategy includes: Determine the sensitivity of each object to be quantized in each quantization strategy, and obtain the sensitivity of each quantization strategy based on the sensitivity of each object to be quantized, wherein the sensitivity of each object to be quantized is obtained based on one or more parameters, including the average trace of the Hessian matrix of the object to be quantized and the loss value of the object to be quantized before and after quantization.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The first quantification strategy is quantitatively evaluated to obtain the quantification evaluation results; If the quantitative evaluation result is unqualified, the first quantitative strategy is rolled back, and the quantitative data type of the object to be quantized in the first quantitative strategy is modified to obtain a second quantitative strategy. The second quantitative strategy is used to quantize the one or more objects to be quantized.

4. The method according to claim 3, characterized in that, The method further includes: The model to be quantized is quantized using the first quantization strategy or the second quantization strategy to obtain the quantized model. The quantized model is sent to the inference system.

5. The method according to claim 3 or 4, characterized in that, The model to be quantized includes multiple network layers. In the first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. In the second quantization strategy, the quantization data type of the same object to be quantized is the same or different in each network layer. The number of objects to be quantized in the second quantization strategy is less than the number of objects to be quantized in the first quantization strategy.

6. The method according to claim 5, characterized in that, The step of performing quantization rollback on the first quantization strategy to obtain the second quantization strategy includes: Obtain multiple rollback plans, each of which includes at least one object to be quantized. The number of objects to be quantized varies between different rollback plans. Determine the quantization data type of each object to be quantized in the first rollback scheme among the multiple rollback schemes, and obtain the third quantization strategy corresponding to the first rollback scheme; The third quantification strategy is quantitatively evaluated to obtain the rollback evaluation result; If the rollback evaluation result is satisfactory, the third quantization strategy will be determined as the second quantization strategy. If the rollback evaluation result is unqualified, the quantization data type of each object to be quantized in the second rollback scheme is determined, and the fourth quantization strategy corresponding to the second rollback scheme is obtained, wherein the number of objects to be quantized in the first rollback scheme is not less than the number of objects to be quantized in the second rollback scheme.

7. The method according to claim 6, characterized in that, The step of determining the quantization data type of each object to be quantized in the first rollback scheme among the plurality of rollback schemes, and obtaining the third quantization strategy corresponding to the first rollback scheme, includes: Based on the objects to be quantized included in the first rollback scheme, and at least one data type that each object to be quantized can be quantized, at least one level quantization strategy for each network layer under the first rollback scheme is obtained. Determine the sensitivity of the quantization strategy for each layer of each network layer, and based on the sensitivity of the quantization strategy for each layer, determine the quantization strategy for each network layer; Based on the quantization strategy of each network layer, a third quantization strategy corresponding to the first backoff scheme is obtained.

8. The method according to any one of claims 1 to 7, characterized in that, The objects to be quantized include linear transformation objects, key-value cache (KV cache) objects, set communication objects, and attention computation objects. The linear transformation objects include operators for performing linear transformations on the data, activation parameters, and weight parameters. The KV cache objects include the K and V vectors in the KV cache. The attention computation objects include operators for matrix multiplication in the attention layer. The set communication objects include communication data during set communication.

9. The method according to any one of claims 6 to 8, characterized in that, The method further includes: If all the rollback evaluation results of each rollback scheme are unqualified, the rollback evaluation results of each rollback scheme are analyzed to obtain analysis results. The analysis results include the quantized object and network layer information whose impact on quantization error is greater than the threshold. Send a manual rollback prompt message to the user's client. The prompt message includes one or more of the following: the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and the analysis result.

10. A model quantization system, characterized in that, The system includes: The acquisition unit is used to acquire one or more objects to be quantized from the model to be quantized, and at least one data type that each object to be quantized can be quantized. A combination unit is configured to determine multiple quantization strategies based on at least one data type that each object to be quantized can be quantized, wherein each quantization strategy includes a quantization data type for each object to be quantized, and the quantization data type is one of at least one data type that the object to be quantized can be quantized. A determining unit is used to determine the sensitivity of each quantization strategy, wherein the sensitivity is used to indicate the degree of accuracy loss of the model to be quantized after quantization using the quantization strategy; A determining unit is configured to obtain a first quantization strategy from the plurality of quantization strategies based on the sensitivity of each quantization strategy, wherein the first quantization strategy is used to quantize the one or more objects to be quantized.

11. The system according to claim 10, characterized in that, The determining unit is used to determine the sensitivity of each object to be quantized in each quantization strategy, and to obtain the sensitivity of each quantization strategy based on the sensitivity of each object to be quantized, wherein the sensitivity of each object to be quantized is obtained based on one or more parameters, such as the average trace of the Hessian matrix of the object to be quantized and the loss value of the object to be quantized before and after quantization.

12. The system according to claim 10 or 11, characterized in that, The system also includes an evaluation unit and a rollback unit. The evaluation unit is used to perform quantitative evaluation on the first quantization strategy and obtain the quantitative evaluation result. The rollback unit is used to roll back the first quantization strategy when the quantization evaluation result is unqualified, modify the quantization data type of the object to be quantized in the first quantization strategy, and obtain a second quantization strategy. The second quantization strategy is used to quantize the one or more objects to be quantized.

13. The system according to claim 12, characterized in that, The system also includes a quantization unit. The quantization unit is used to quantize the model to be quantized using the first quantization strategy or the second quantization strategy to obtain the quantized model. The quantization unit is used to send the quantized model to the inference system.

14. The system according to claim 12 or 13, characterized in that, The model to be quantized includes multiple network layers. In the first quantization strategy, the quantization data type of the same object to be quantized is the same in each network layer. In the second quantization strategy, the quantization data type of the same object to be quantized is the same or different in each network layer. The number of objects to be quantized in the second quantization strategy is less than the number of objects to be quantized in the first quantization strategy.

15. The system according to claim 14, characterized in that, The rollback unit is used to acquire multiple rollback schemes, each rollback scheme including at least one object to be quantized, and the number of objects to be quantized varies in different rollback schemes; The rollback unit is used to determine the quantization data type of each object to be quantized in the first rollback scheme among the multiple rollback schemes, and to obtain the third quantization strategy corresponding to the first rollback scheme; The rollback unit is used to perform quantitative evaluation on the third quantization strategy to obtain rollback evaluation results; The rollback unit is used to determine the third quantization strategy as the second quantization strategy if the rollback evaluation result is qualified. The rollback unit is used to determine the quantization data type of each object to be quantized in the second rollback scheme when the rollback evaluation result is unqualified, and to obtain the fourth quantization strategy corresponding to the second rollback scheme, wherein the number of objects to be quantized in the first rollback scheme is not less than the number of objects to be quantized in the second rollback scheme.

16. The system according to claim 15, characterized in that, The rollback unit is used to obtain at least one level quantization strategy for each network layer under the first rollback scheme based on the objects to be quantized included in the first rollback scheme and the data type that each object to be quantized can be quantized. The backoff unit is used to determine the sensitivity of the quantization strategy at each level of each network layer, and to determine the quantization strategy for each network layer based on the sensitivity of the quantization strategy at each level. The backoff unit is used to obtain a third quantization strategy corresponding to the first backoff scheme based on the quantization strategy of each network layer.

17. The system according to any one of claims 10 to 16, characterized in that, The objects to be quantized include linear transformation objects, key-value cache (KV cache) objects, set communication objects, and attention computation objects. The linear transformation objects include operators for performing linear transformations on the data, activation parameters, and weight parameters. The KV cache objects include the K and V vectors in the KV cache. The attention computation objects include operators for matrix multiplication in the attention layer. The set communication objects include communication data during set communication.

18. The system according to any one of claims 10 to 17, characterized in that, The system also includes a prompting unit. The prompting unit is used to analyze the rollback evaluation results corresponding to each rollback scheme when all rollback evaluation results of each rollback scheme are unqualified, and obtain analysis results. The analysis results include the quantized object and network layer information whose impact on quantization error is greater than the threshold. The prompting unit is used to send a prompt message for manual rollback to the client. The prompt message includes one or more of the following: the third quantization strategy corresponding to each rollback scheme, the rollback evaluation result corresponding to each third quantization strategy, and the analysis result.

19. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 8.

20. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the operational steps of the method as described in any one of claims 1 to 9.

21. A chip, characterized in that, The chip includes a power supply unit and a processing unit. The power supply unit supplies power to the processing unit, enabling the processing unit to perform the operation steps of the method as described in any one of claims 1 to 9.

22. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 9.

23. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the operational steps of the method as described in any one of claims 1 to 9.

Images