AI model quantification method and device and related equipment

Abstract

The invention discloses an AI model quantification method and device and related equipment, and relates to the technical field of artificial intelligence. Dividing a plurality of weights in the AI model into a first weight set and a second weight set, wherein the sensitivity of important weights included in the first weight set to quantization is higher than the sensitivity of weights in the second weight set to quantization; noise is added to the values of the weights, belonging to the second weight set, in the AI model, fine adjustment is conducted on the AI model through the data set, and in the process of fine adjustment of the AI model, the values of the weights, belonging to the first weight set, will be updated, and the values of the weights, belonging to the second weight set, will not be updated; and quantizing the values of the weights belonging to the second weight set in the AI model. Thus, only the value of the weight with low sensitivity to quantization in the AI model is quantized, the noise is injected into the weight with low sensitivity in the AI model, and the AI model is finely adjusted, so that the reasoning precision of the quantized AI model can still be kept at a high level.

Description

[0001] This application claims priority to Russian patent application filed on August 15, 2024, with application number RU2024123479 and entitled "One Quantization-friendly Fine-tuning solution based on training of Salient Weights and Noise Injection to other Weights", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to an AI model quantification method, apparatus and related equipment. Background Technology

[0003] With the development of artificial intelligence (AI) technology, AI models such as large language models (LLM) not only have high inference performance, but also include a large number of parameters, such as hundreds of billions of parameters in AI models.

[0004] Typically, AI models can reduce their data size through compression techniques such as quantization, thereby reducing the memory space required for the AI ​​model to run. Quantization refers to converting high-precision parameters in the AI ​​model into low-precision parameters, such as converting weight values ​​and 32-bit floating-point numbers (high precision) into 8-bit integers (low precision).

[0005] However, in real-world applications, the accuracy of AI models often decreases significantly after quantization, thus reducing their inference performance. For example, an AI model with an accuracy of 78.05% before quantization might only achieve 69.53% after quantization. Summary of the Invention

[0006] This application provides an AI model quantization method, aiming to maintain a high level of inference accuracy in the quantized AI model. Furthermore, this application also provides an AI model quantization apparatus, a computing device, a computer-readable storage medium, and a computer program product.

[0007] Firstly, this application provides an AI model quantization method, which can be executed by a corresponding AI model quantization device. Specifically, the AI ​​model quantization device divides multiple weights in the AI ​​model into a first weight set and a second weight set. These multiple weights can be all weights in the AI ​​model or only some weights. Furthermore, the first weight set includes important weights, whose sensitivity to quantization is higher than that of the weights in the second weight set. This sensitivity indicates the deviation between the activation value generated by the quantized weight and the activation value generated by the weight before quantization. For example, the sensitivity can specifically be this deviation, or a value calculated based on this deviation. In practical applications, all weights in the first weight set... Both can be important weights, or the first weight set can include both important weights and weights with relatively low sensitivity to quantization. Then, the AI ​​model quantization device adds noise to the values ​​of the weights in the AI ​​model that belong to the second weight set. The added noise can be used to simulate the noise brought about by quantization of the AI ​​model. Furthermore, the AI ​​model quantization device uses the dataset to fine-tune the AI ​​model after adding noise. During the fine-tuning of the AI ​​model, the values ​​of the weights in the AI ​​model that belong to the first weight set are updated, while the values ​​of the weights in the AI ​​model that belong to the second weight set are not updated (i.e., these weights are frozen). Finally, the AI ​​model quantization device can quantize the values ​​of the weights in the second weight set in the fine-tuned AI model.

[0008] Because the deviation between the calculated activation values ​​and the values ​​of some weights in an AI model is small before and after quantization, the impact of quantization on the inference accuracy of the AI ​​model is minimal. However, the deviation between the calculated activation values ​​and the values ​​of other weights in the AI ​​model is large, which can easily lead to a significant decrease in the inference accuracy of the AI ​​model. Therefore, during the quantization process, the AI ​​model quantization device only quantizes the values ​​of weights belonging to the second weight set that have low sensitivity to quantization, while excluding the values ​​of important weights with high sensitivity. This effectively avoids a significant decrease in the inference accuracy of the quantized AI model, meaning that the inference accuracy of the quantized AI model can still be maintained at a high level. Furthermore, by injecting noise into the low-sensitivity weights of the AI ​​model and fine-tuning the AI ​​model before quantization, the noise interference introduced by quantization can be balanced. This allows the inference accuracy of the quantized AI model to approximate that of the AI ​​model before quantization, further ensuring that the inference accuracy of the quantized AI model reaches a high level.

[0009] In one possible implementation, when the AI ​​model quantization device adds noise to the values ​​of weights belonging to the second weight set in the AI ​​model, it can specifically add Gaussian noise to the values ​​of weights belonging to the second weight set in the AI ​​model. Thus, by using normally distributed Gaussian noise, the quantization noise generated by AI model quantization can be better simulated.

[0010] In one possible implementation, when the AI ​​model quantization device adds noise to the weights belonging to the second weight set in the AI ​​model, it can first estimate the quantized values ​​of the weights in the second weight set, and then add noise to the weight values ​​based on the distribution of the quantized weight values. In this way, the noise added by the AI ​​model quantization device to the weight values ​​belonging to the second weight set in the AI ​​model can better simulate the quantization noise generated by the AI ​​model after actual quantization processing, thereby helping to improve the inference accuracy of the quantized AI model.

[0011] In one possible implementation, the second weight set includes multiple weight columns. When the AI ​​model quantization device adds noise to the weight values ​​in the second weight set based on the quantized value distribution of the weights in the second weight set, specifically, it can determine a quantization scaling factor for each of the multiple weight columns based on the quantized value distribution of the weights in the second weight set. The quantization scaling factor is used to determine the magnitude of the noise added to the weight column. Then, the AI ​​model quantization device adds noise to the weight values ​​in each weight column according to the quantization scaling factor corresponding to each weight column. Thus, after the AI ​​model quantization device 200 adds Gaussian noise to the weight values ​​belonging to the second weight set in the AI ​​model, the distribution of the weight values ​​in the second weight set can approach the quantized value distribution of the weights. Therefore, the added Gaussian noise can better simulate the quantization noise brought about by quantizing the weights in the second weight set.

[0012] In one possible implementation, when the AI ​​model quantization device divides multiple weights in the AI ​​model into a first weight set and a second weight set, it may specifically evaluate the sensitivity of each weight in the AI ​​model to quantization, and divide the multiple weights into the first weight set and the second weight set based on the sensitivity of each weight to quantization. In this way, the AI ​​model quantization device divides multiple weights according to the identified sensitivity of each weight to quantization, which can improve the accuracy of weight division. Therefore, after quantizing weights with lower sensitivity, the inference accuracy of the quantized AI model can still be maintained at a high level.

[0013] In one possible implementation, when the AI ​​model quantization device divides multiple weights into a first weight set and a second weight set based on the sensitivity of each weight to quantization, it may specifically allocate the weight columns of the first part of the weight matrix corresponding to the multiple weights into the first weight set and the weight columns of the second part of the weight matrix into the second weight set based on a first ratio and the sensitivity of each weight to quantization. In the first weight set, each weight column has a weight whose sensitivity to quantization is higher than the total sensitivity of all weights in each weight column of the second weight set. Furthermore, the ratio between the number of weight columns in the first weight set and the number of weight columns in the second weight set is the first ratio. Alternatively, the AI ​​model quantization device may divide the multiple weights into the first weight set based on a second ratio and the sensitivity of each weight to quantization. The first weight set includes a second weight set, where the weights in the first weight set are all at least as sensitive to quantization as the weights in the second weight set. The ratio between the number of weights in the first weight set and the number of weights in the second weight set is a second ratio. Alternatively, the AI ​​model quantization device can assign weights with a quantization sensitivity greater than a first threshold to the first weight set and weights with a quantization sensitivity no greater than the first threshold to the second weight set. Or, the AI ​​model quantization device can assign weight columns of a first type from the weight matrix corresponding to multiple weights to the first weight set and weight columns of a second type from the weight matrix to the second weight set, where weights in the first type of weight column have a quantization sensitivity greater than the second threshold, but no weights in the second type of weight column have a quantization sensitivity greater than the second threshold. Thus, the AI ​​model quantization device can divide multiple weights into two different weight sets through the above various implementation methods, so that the inference accuracy of the quantized AI model can be maintained at a high level based on these two weight sets.

[0014] In one possible implementation, when evaluating the sensitivity of each weight among multiple weights in an AI model to quantization, the AI ​​model quantization device may first obtain the first value and input activation value of each weight before quantization, and estimate the second value of each weight after quantization. Then, the AI ​​model quantization device calculates the difference between the first and second values ​​of each weight, and calculates the sensitivity of each weight to quantization based on the difference and the input activation value. For example, the AI ​​model quantization device may use the product of the difference and the activation value as the sensitivity.

[0015] Secondly, this application provides an AI model quantization device, which includes a partitioning module for partitioning multiple weights in the AI ​​model into a first weight set and a second weight set. The first weight set includes important weights, and the sensitivity of important weights to quantization is higher than that of the weights in the second weight set. The sensitivity is used to indicate the deviation of the activation value generated by the weight after quantization from the activation value generated before quantization. An adding module is used to add noise to the values ​​of the weights in the AI ​​model that belong to the second weight set. A fine-tuning module is used to fine-tune the AI ​​model after adding noise using a dataset. During the fine-tuning process, the values ​​of the weights in the AI ​​model that belong to the first weight set are updated, while the values ​​of the weights in the AI ​​model that belong to the second weight set are not updated. A quantization module is used to quantize the values ​​of the weights in the AI ​​model that belong to the second weight set after fine-tuning.

[0016] In one possible implementation, a module is added to add Gaussian noise to the values ​​of weights in the AI ​​model that belong to the second weight set.

[0017] In one possible implementation, a module is added for: estimating the quantized values ​​of the weights in the second weight set; and adding noise to the values ​​of the weights in the second weight set based on the distribution of the quantized values ​​of the weights in the second weight set.

[0018] In one possible implementation, the second weight set includes multiple weight columns, and the adding module is used to: determine the quantization scaling factor corresponding to each weight column in the multiple weight columns according to the distribution of the weight values ​​in the second weight set after quantization, the quantization scaling factor being used to determine the amplitude of the noise added to the weight column; and add noise to the value of the weight in each weight column according to the quantization scaling factor corresponding to each weight column.

[0019] In one possible implementation, a partitioning module is used to: evaluate the sensitivity of each of a plurality of weights in the AI ​​model to quantization; and partition the plurality of weights into a first set of weights and a second set of weights based on the sensitivity of each weight to quantization.

[0020] In one possible implementation, the partitioning module is configured to: assign weight columns from a first portion of the weight matrix corresponding to multiple weights to a first weight set, and assign weight columns from a second portion of the weight matrix to a second weight set, based on a first ratio and the sensitivity of each weight to quantization; wherein each weight column in the first weight set has a weight whose sensitivity to quantization is higher than the total sensitivity of all weights in each weight column of the second weight set to quantization, and the ratio between the number of weight columns in the first weight set and the number of weight columns in the second weight set is the first ratio; or, based on the second ratio and the sensitivity of each weight to quantization, partition the multiple weights into a first weight set and a second weight set, wherein the weights in the first weight set... The sensitivity of each weight to quantization is no less than the sensitivity of the weights in the second weight set to quantization, and the ratio between the number of weights in the first weight set and the number of weights in the second weight set is the second ratio; or, weights with a sensitivity to quantization greater than the first threshold are assigned to the first weight set, and weights with a sensitivity to quantization no greater than the first threshold are assigned to the second weight set; or, weight columns of the first type in the weight matrix corresponding to multiple weights are assigned to the first weight set, and weight columns of the second type in the weight matrix are assigned to the second weight set, where there are weights in the first type of weight column with a sensitivity to quantization greater than the second threshold, and no weights in the second type of weight column have a sensitivity to quantization greater than the second threshold.

[0021] In one possible implementation, the partitioning module is used to: obtain the first value and input activation value of each weight before quantization; estimate the second value of each weight after quantization; calculate the difference between the first value and the second value of each weight; and calculate the sensitivity of each weight to quantization based on the difference and the input activation value.

[0022] The AI ​​model quantization device provided in the second aspect corresponds to the AI ​​model quantization method provided in the first aspect. Therefore, the technical effects of the second aspect and any implementation thereof can be found in the relevant descriptions of the technical effects of the first aspect and the corresponding implementation thereof, and will not be repeated here.

[0023] Thirdly, this application provides a computing device including a processor and a memory; wherein the memory is used to store instructions, and the processor executes the instructions stored in the memory to perform the operation steps of the AI ​​model quantization method described in the first aspect and any implementation thereof.

[0024] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operation steps of the AI ​​model quantization method described in the first aspect or any implementation thereof.

[0025] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the AI ​​model quantization method described in the first aspect or any implementation thereof.

[0026] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0027] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided in this application;

[0028] Figure 2 A flowchart illustrating an AI model quantization method provided in this application;

[0029] Figure 3 A schematic diagram to predict the sensitivity of each weight to quantization;

[0030] Figure 4 This is a schematic diagram illustrating the reordering of multiple weight columns in a weight matrix.

[0031] Figure 5 This is a schematic diagram illustrating noise injection and matrix calculation in an AI model.

[0032] Figure 6 This is a diagram illustrating the results obtained from testing two AI models on multiple datasets.

[0033] Figure 7 A schematic diagram of the structure of an AI model quantization device provided in this application;

[0034] Figure 8 This is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0035] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, various non-limiting embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained based on the embodiments in this application and based on the above content are within the scope of protection of this application.

[0036] See Figure 1 The diagram illustrates the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes multiple computing nodes. For ease of understanding, Figure 1 The following description uses compute nodes 101 to 104 as an example. Multiple compute nodes can be interconnected via a bus, such as a peripheral component interconnect express (PCIe) bus or an InfiniBand bus. Alternatively, multiple compute nodes can be interconnected via a network, such as a communication network including at least one switch.

[0037] The computing node can be a node with model training capabilities. For example, the computing node can be an accelerator card, such as a deep-learning processing unit (DPU), data processing unit (DPU), graphics processing unit (GPU), neural-network processing unit (NPU), or tensor processing unit (TPU), or other types of accelerator cards. Alternatively, the computing node can be a general-purpose processor, such as a central processing unit (CPU). Or, the computing node can also include a CPU and an accelerator card computing device. This application does not limit the specific implementation of the computing node.

[0038] exist Figure 1 In the data processing system 10 shown, the AI ​​model can be trained using computing nodes 101 and 104. For example, the AI ​​model can be a large language model Meta AI (LLaMA), a bidirectional encoder representations from transformers (BERT), or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models such as GPT-4, etc., without limitation.

[0039] In practical applications, trained AI models are typically deployed to other resource environments, such as the user's hardware environment. Due to the large amount of data in AI models (e.g., hundreds of billions of parameters) and the resource limitations (such as memory) in user hardware environments, the data processing system 10 usually quantizes the weights and activation values ​​after training. This can be done using strategies like W8A8 or W4A16. W8A8 quantizes high-precision weights and activation values ​​(represented as 32-bit floating-point numbers) into 8-bit integers. W4A16 quantizes high-precision weights (represented as 32-bit floating-point numbers) into 4-bit integers and high-precision activation values ​​(represented as 32-bit floating-point numbers) into 16-bit floating-point numbers. In this way, in the quantized AI model, each weight value and activation value is represented by fewer bits, which can effectively reduce the memory and computing resources required by the AI ​​model during operation, so that the operation requirements of the AI ​​model can still be met even when the user's hardware environment resources are limited.

[0040] However, the accuracy of AI models is usually achieved based on high-precision parameters. This means that after quantizing the weights and activation values ​​in the AI ​​model, there is a discrepancy between the original and quantized weight values. This discrepancy can easily lead to a significant decrease in the inference accuracy of the AI ​​model. To address this, the data processing system 10 typically uses the parameter efficient fine-tuning (PERT) method to fine-tune the quantized AI model to improve its inference accuracy. However, in practical applications, the inference accuracy of the fine-tuned AI model still differs significantly from that of the AI ​​model before quantization.

[0041] Based on this, the data processing system 10 provided in this application may also include an AI model quantization device 200, which is used to quantify the AI ​​model and ensure that the inference accuracy of the quantized AI model remains at a high level.

[0042] In specific implementation, before quantizing the AI ​​model, the AI ​​model can be run through computing nodes 101 to 104. For example, the AI ​​model quantization device 200 can schedule computing nodes 101 to 104 to run the AI ​​model using a specified dataset. Furthermore, the AI ​​model quantization device 200 can divide multiple weights in the AI ​​model into weight set 1 and weight set 2. These multiple weights can be all weights in the AI ​​model or only some weights. Weight set 1 includes salient weights, which have a higher sensitivity to quantization than the individual weights in weight set 2. The sensitivity of each weight to quantization indicates the deviation of the activation value generated by the weight after quantization from the activation value generated by the weight before quantization. Generally, the larger the deviation (i.e., the larger the absolute value of the difference between the two activation values), the higher the sensitivity. The activation value generated by the weight's value refers to the activation value calculated based on that weight. Then, the AI ​​model quantization device 200 adds noise to the weights belonging to weight set 2 in the AI ​​model. This noise, such as Gaussian noise, can be used to simulate the quantization error introduced after subsequent quantization of the AI ​​model. Next, the AI ​​model quantization device 200 can fine-tune the AI ​​model after adding noise. Specifically, it fine-tunes the weights belonging to weight set 1 in the AI ​​model. That is, during the fine-tuning process, the weights belonging to weight set 1 in the AI ​​model are updated, while the weights belonging to weight set 2 are not updated. Finally, the AI ​​model quantization device 200 can quantize the weights belonging to weight set 2 in the fine-tuned AI model, such as converting the weight values ​​from 32-bit floating-point to 8-bit integers.

[0043] For the weights belonging to weight set 1 in the AI ​​model, the deviation between the calculated activation values ​​before and after quantization is small. This means the activation values ​​remain essentially unchanged after quantization, minimizing the impact of quantization on the inference accuracy of the AI ​​model. However, for the important weights identified in the AI ​​model, the deviation between the calculated activation values ​​before and after quantization is large. This significant deviation can lead to a substantial decrease in the inference accuracy of the AI ​​model. Therefore, during the quantization process, the AI ​​model quantization device 200 only quantizes the weights belonging to weight set 2 that have low sensitivity to quantization, excluding the highly sensitive important weights. This effectively avoids a significant decrease in the inference accuracy of the quantized AI model, ensuring that the inference accuracy remains at a high level. Furthermore, before quantizing the AI ​​model, the AI ​​model quantization device 200 injects noise into the AI ​​model's less sensitive weights (i.e., weights belonging to weight set 2) and fine-tunes the AI ​​model, which can balance the noise interference brought about by quantization. This makes the inference accuracy of the quantized AI model approximately the same as that of the AI ​​model before quantization, thereby further ensuring that the inference accuracy of the quantized AI model can reach a high level.

[0044] For example, the AI ​​model quantization device 200 described above can be implemented by software or hardware.

[0045] In the first example, the AI ​​model quantization device 200 can be implemented in software. Specifically, the AI ​​model quantization device 200 can be code running on a computing instance, such as code running on a physical device, virtual machine, or container. Furthermore, when the AI ​​model quantization device 200 runs on at least one physical device, that physical device can be connected to multiple computing nodes via a network or bus. This allows the AI ​​model quantization device 200 on the physical device to communicate with multiple computing nodes and quantize the AI ​​model based on this connection.

[0046] In the second example, when implemented in hardware, the AI ​​model quantization device 200 can be implemented through a computing node in the data processing system 10, or through at least one physical device including a processor, such as a server. In this case, the AI ​​model quantization device 200 can be connected to multiple computing nodes via a bus, such as... Figure 1As shown; alternatively, the AI ​​model quantization device 200 can be connected to multiple computing nodes via a network. The processor can be a CPU, application-specific integrated circuit (ASIC), programmable logic device (PLD), complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL), system-on-chip (SoC), software-defined infrastructure (SDI) chip, artificial intelligence (AI) chip, or DPU, or any combination of the above processors. Furthermore, the AI ​​model quantization device 200 can include one or more processors, and the types of processors can be one or more. The specific number and types of processors can be determined according to the actual application's business requirements; this embodiment does not limit this.

[0047] It is worth noting that the above Figure 1 The data processing system 10 shown is merely an illustrative example and is not intended to limit the scope of the system. For instance, other possible data processing systems may include more or fewer computing nodes, or may include nodes with other functionalities, such as nodes for storing data or nodes for managing multiple computing nodes. Similarly, other possible communication systems may include a client, which could be program code deployed on a user-side terminal device or a web browser, used for human-computer interaction, such as obtaining quantization (and training) instructions for the AI ​​model specified by the user through the client, and providing / specifying datasets for running / training the AI ​​model.

[0048] For ease of understanding, the embodiments for quantizing AI models provided in this application are described below with reference to the accompanying drawings.

[0049] See Figure 2 , Figure 2 This is a flowchart illustrating an exemplary AI model quantization method provided in an embodiment of this application. Figure 2 The AI ​​model quantization method shown can be applied to Figure 1The data processing system 10 shown can be applied to other possible data processing systems. For ease of understanding and description, it will be used in the following example. Figure 1 The data processing system 10 shown is illustrated using computing nodes 101 to 104 as an example to illustrate the operation of an AI model.

[0050] like Figure 2 As shown, the AI ​​model quantization method can specifically include the following steps.

[0051] S201: AI model quantization device 200 provides a dataset to multiple computing nodes and instructs the multiple computing nodes to run AI models based on the dataset.

[0052] In practical applications, the AI ​​model can be pre-trained on multiple computing nodes (including computing nodes 101 to 104), and the AI ​​model quantization device 200 can further quantize the trained AI model to reduce the amount of data in the AI ​​model. Alternatively, the user can provide the AI ​​model to be quantized to the data processing system 10 through a client and request the data processing system 10 to quantize the AI ​​model.

[0053] At this time, the AI ​​model quantization device 200 can instruct multiple computing nodes to run the AI ​​model to be quantized and provide a dataset to these computing nodes. This dataset serves as the input data for the AI ​​model during runtime. For example, the dataset could contain 512 random statements. During the execution of the AI ​​model, computing nodes 101 to 104 can calculate the activation value of the current network layer based on the weight values ​​of each network layer in the AI ​​model, and output the activation value of the current network layer, or data further calculated based on that activation value, to the next network layer as its input. The activation value of the first network layer is the input data of the AI ​​model.

[0054] S202: The AI ​​model quantization device 200 divides multiple weights in the AI ​​model into weight set 1 and weight set 2. Weight set 1 includes important weights, which have a higher sensitivity to quantization than the weights in weight set 2. This sensitivity is used to indicate the deviation between the activation value generated by the weight after quantization and the activation value generated before quantization.

[0055] Generally, the impact of different weight values ​​on the inference accuracy of an AI model can vary. That is, changing the values ​​of some weights in an AI model may have a small impact on the inference accuracy; for example, changing the values ​​of those weights may not affect the inference accuracy. However, changing the values ​​of other weights in an AI model may have a significant impact on the inference accuracy; for example, changing the values ​​of those weights may cause a noticeable decrease in the inference accuracy.

[0056] Therefore, for example, before quantizing the AI ​​model, the AI ​​model quantization device 200 can first identify the weights in the AI ​​model that are relatively sensitive to quantization (i.e., have high sensitivity to quantization). Here, weight sensitivity to quantization means that the value of the weight will have a significant impact on the inference accuracy of the AI ​​model after quantization. In this embodiment, the magnitude of the deviation between the activation values ​​generated by the weight before and after quantization can be used to determine whether the change in the weight value will have a significant impact on the inference accuracy of the AI ​​model. Then, the AI ​​model quantization device 200 can divide the multiple weights into weight set 1 and weight set 2 according to the sensitivity of each weight to quantization.

[0057] The following sections describe how the AI ​​model quantization device 200 evaluates the sensitivity of each weight to quantization and how it divides multiple weights based on the sensitivity of each weight.

[0058] 1. The AI ​​model quantization device 200 evaluates the sensitivity of each of the multiple weights in an AI model to quantization.

[0059] In one possible implementation, the AI ​​model quantization device 200 can acquire the values ​​of multiple weights in the AI ​​model before quantization, hereinafter referred to as the first values. For example, the AI ​​model quantization device 200 can read the first values ​​corresponding to each of the multiple weights from computing nodes 101 to 104. The multiple weights in the AI ​​model can be all the weights in the AI ​​model or only some of the weights; this is not limited. During the process of running the AI ​​model based on the dataset, the AI ​​model quantization device 200 also acquires the input activation value of the weight, which is the activation value used as the input to the network layer where the weight resides. For example, the input activation value can be read from computing nodes 101 to 104. Furthermore, the AI ​​model quantization device 200 can also predict the possible values ​​of each of the multiple weights after quantization, hereinafter referred to as the second values. For example, the AI ​​model quantization device 200 can calculate the second values ​​of each weight after quantization according to quantization rules. Then, the AI ​​model quantization device 200 calculates the difference between the first and second values ​​of each weight, and calculates the sensitivity of the weight to quantization based on the difference and the input activation value. For example, the AI ​​model quantization device 200 can use the product of the difference between the first and second values ​​and the weight and the input activation value as the sensitivity of the weight to quantization.

[0060] For example, for a weight matrix in an AI model, the AI ​​model quantization device 200 can calculate the sensitivity of each weight based on the following formula (1).

[0061]

[0062] Among them, s j W represents the sensitivity of the weight in the j-th column of the weight matrix; :,j Q(W) represents the first value of the weight in the j-th column of the weight matrix. :,j ) represents the second value of the weight of the j-th column after uniform symmetric quantization; ||W :,j -Q(W :,j )|| ∞ It is the absolute value of the difference between the first and second values ​​of the weight in the j-th column of the weight matrix; The input activation value corresponding to the weight in the j-th column of the weight matrix; ||X|| ∞ γ represents the input activation value corresponding to the entire weight matrix; γ is the norm order coefficient, which can take values ​​such as 0.5, 1, or 2.

[0063] Taking the calculation of the sensitivity of weights in a network layer of an AI model to quantization as an example, the multiple weights of this network layer can form a weight matrix, and the first value of each weight in this weight matrix can be as follows: Figure 3 As shown. Assuming the quantization step size is 1, the AI ​​model quantization device 200 can predict the second value of each weight in the weight matrix after quantization, as shown. Figure 3 As shown. Then, the AI ​​model quantization device 200 can calculate the absolute value of the difference between the first value and the second value of each weight in the weight matrix, and further calculate the product between the absolute value of the difference corresponding to each weight in the weight matrix and the input activation value of the weight matrix, and use this product as the sensitivity of the weight to quantization, such as... Figure 3 As shown. For example, the weights in the first row and first column of the weight matrix have a sensitivity of 0.5 to quantization, and the weights in the first row and third column have a sensitivity of 2.4 to quantization.

[0064] It is understood that the above implementation of the method for estimating the sensitivity of each weight to quantization is only an implementation example and is not intended to limit the scope.

[0065] For example, in other possible implementations, the AI ​​model quantization device 200 can first calculate the activation value generated by the first value and the activation value generated by the second value of each weight, then calculate the difference between the two activation values, and use the difference or the absolute value of the difference as the sensitivity of the weight to quantization.

[0066] For example, in other possible implementations, when the AI ​​model quantization device 200 calculates the sensitivity of the weight to quantization based on the difference between the first and second values ​​of the weight and the input activation value, it may further calculate the product between the difference and the input activation value, such as normalizing or scaling the product, and use the result of the further calculation as the sensitivity of the weight to quantization, etc., without limitation.

[0067] 2. The AI ​​model quantization device 200 divides multiple weights into weight set 1 and weight set 2 according to the sensitivity of each weight to quantization. In this embodiment, the following non-limiting implementation examples are provided.

[0068] In the first implementation example, the AI ​​model quantization device 200 can divide multiple weights into two weight sets based on a pre-set first threshold, using individual weights as the granularity. Specifically, the AI ​​model quantization device 200 can compare the sensitivity of each weight to quantization with the first threshold, and assign weights with a sensitivity greater than the first threshold to weight set 1, and weights with a sensitivity less than or equal to the first threshold to weight set 2. In this case, all weights in weight set 1 are important weights, i.e., weights with high sensitivity, and the sensitivity of each weight in weight set 1 to quantization is higher than that of each weight in weight set 2. In practical applications, the aforementioned first threshold can be set by the user or technical personnel, etc.

[0069] In the second implementation example, the AI ​​model quantization device 200 can divide multiple weights into two weight sets based on a pre-set second ratio, using individual weights as the granularity. Specifically, the AI ​​model quantization device 200 can sort multiple weights according to their sensitivity to quantization, such as sorting them in descending or ascending order of sensitivity. Then, the AI ​​model quantization device 200 can divide the sorted weights into two weight sets, namely weight set 1 and weight set 2, according to the second ratio. Each weight in weight set 1 has a quantization sensitivity greater than or equal to that in weight set 2. That is, all weights in weight set 1 are important weights. In practical applications, the second ratio used to divide the multiple weights can be set by the user or technicians.

[0070] In a third implementation example, the AI ​​model quantization device 200 can divide multiple weights into two weight sets based on a pre-set second threshold, using a column of weights in the weight matrix as the granularity. Specifically, the AI ​​model quantization device 200 can traverse at least one weight matrix (the weights included in the at least one weight matrix are the aforementioned multiple weights), and for each column of weights, the AI ​​model quantization device 200 can compare the sensitivity of each weight in that column to quantization with the second threshold to determine whether there are any weights in that column whose sensitivity to quantization is greater than the second threshold. If so, the AI ​​model quantization device 200 can classify the weights included in that column as a first type of weight column into weight set 1; if not, the AI ​​model quantization device 200 can classify the weights included in that column as a second type of weight column into weight set 2. In this way, the AI ​​model quantization device 200 can divide the multiple weight columns included in at least one weight matrix into two weight sets. At this time, weight set 1 includes not only important weights, but also non-important weights (i.e., weights whose sensitivity to quantization is lower than the second threshold).

[0071] In the fourth implementation example, the AI ​​model quantization device 200 can divide multiple weights into two weight sets based on a pre-set first ratio, using a column of weights in the weight matrix as the granularity. Specifically, the AI ​​model quantization device 200 can first traverse at least one weight matrix to determine the maximum sensitivity of the weights in each weight column to quantization, and then, based on the first ratio and the maximum sensitivity corresponding to each weight column, divide the multiple weight columns included in the at least one weight matrix into two weight sets, namely weight set 1 and weight set 2, as follows. Figure 4 As shown on the left. In weight set 1, the maximum sensitivity value corresponding to each weight column is greater than or equal to the maximum sensitivity value corresponding to each weight column in weight set 2. Furthermore, the ratio between the number of weight columns in weight set 1 and the number of weight columns in weight set 2 is this first ratio. In this case, weight set 1 may include not only important weights but also unimportant weights.

[0072] For example, regarding such Figure 3 The weight matrix shown has a maximum sensitivity value of 0.5 for the first column, 0.8 for the second column, 2.4 for the third column, and 1.5 for the fourth column. Assuming a first ratio of 1, the AI ​​model quantization device 200 can allocate the weight values ​​of the third and fourth columns to weight set 1 and the weight values ​​of the first and second columns to weight set 2 based on the maximum sensitivity value of each weight column. At this time, the number of weight columns in weight set 1 is 2, the number of weight columns in weight set 2 is 2, and the ratio between the number of weight columns in weight set 1 and the number of weight columns in weight set 2 is 1 (i.e., equal to the first ratio).

[0073] In practical applications, quantizing the weights in an AI model introduces quantization noise, which can easily reduce the inference accuracy of the AI ​​model. Therefore, in this embodiment, the AI ​​model quantization device 200 can inject noise into the AI ​​model and fine-tune it before quantization, and then quantize the AI ​​model after fine-tuning it. This balances the noise interference from subsequent quantization by injecting noise and fine-tuning the AI ​​model. The following continues in conjunction with... Figure 2 Steps S203 to S205 describe the process of first injecting noise, fine-tuning the AI ​​model, and then quantizing the AI ​​model based on the weight set 2.

[0074] S203: AI model quantization device 200 adds noise to the values ​​of weights belonging to weight set 2 in the AI ​​model.

[0075] For example, when the AI ​​model quantization device 200 adds noise to the values ​​of the weights belonging to the weight set 2, it can specifically add Gaussian noise. Gaussian noise, also known as white noise or random noise, is a type of interference noise that conforms to a Gaussian (normal) distribution.

[0076] In one possible implementation, assuming the AI ​​model quantization device 200 divides multiple weight columns into weight set 2 at the granularity of weight columns, the AI ​​model quantization device 200 reorders the channels of the weight columns in each weight matrix. The reordered weight columns can form submatrices corresponding to weight set 1 and weight set 2. For example, the AI ​​model quantization device 200 can reorder multiple weight columns in the weight matrix as follows: Figure 4 The two sub-matrices shown on the right. Furthermore, the AI ​​model quantization device 200 also records the correspondence between the position of the sorted weight column and its original position in the weight matrix.

[0077] Then, the AI ​​model quantization device 200 can randomly sample elements from a standard normal distribution for the submatrix corresponding to weight set 2, and generate a noise matrix based on the sampled elements. The generated noise matrix has the same shape as the submatrix. Next, the AI ​​model quantization device 200 can generate a diagonal matrix for the noise matrix using the quantization step size, and scale the noise matrix using this diagonal matrix. Thus, by adding the scaled noise matrix to the submatrix corresponding to weight set 2, the AI ​​model quantization device 200 can add Gaussian noise to the weights belonging to weight set 2 in the AI ​​model.

[0078] For example, the AI ​​model quantization device 200 can add Gaussian noise to the weights belonging to the weight set 2 in the AI ​​model by the following formula (2).

[0079]

[0080] Where Ψ refers to the values ​​of multiple weights in the AI ​​model after adding Gaussian noise; W [:,salient] The submatrix corresponding to weight set 1; is W. [:,non-salient] Ω is the submatrix corresponding to weight set 2; Δ is the quantization step size; diag(Δ) is the diagonal matrix generated based on the quantization step size; Ω is the noise matrix; This refers to adding Gaussian noise to the weight values ​​in the submatrix corresponding to weight set 2.

[0081] Furthermore, for each weight column in the submatrix corresponding to weight set 2, a suitable quantization scaling factor can be determined. This quantization scaling factor is the factor in the quantization step size Δ used to add noise to a weight column, and can be used to determine the amplitude of the noise to be added to the weight column. Thus, the AI ​​model quantization device 200 quantizes the submatrix W corresponding to weight set 2. [:,non-salient] After adding Gaussian noise, the distribution of weight values ​​in weight set 2 can approximate the distribution of quantized weight values. Therefore, the added Gaussian noise can better simulate the quantization noise introduced after quantizing the weights in weight set 2.

[0082] For example, the AI ​​model quantization device 200 can determine a suitable quantization scaling factor for each weight column by the following formula (3).

[0083]

[0084] Where, Δ i α is the quantization scaling factor in the i-th column of the quantization step, used to add noise to the weight column in the i-th column of the submatrix corresponding to weight set 2; b is the bit width, and the smaller the bit width, the larger the value of the quantization scaling factor, thus generating more noise; i These are quantization parameters.

[0085] In practical applications, the AI ​​model quantization device 200 can determine the quantization parameter α by optimizing the weight error through linear search. i The value of α. For example, the AI ​​model quantization device 200 can calculate the difference between the first and second values ​​of the weights in the weight column, and determine the quantization parameter α based on this difference through a linear search. i So that the determined quantization parameter α i It can best approximate this difference. Thus, based on this quantization parameter α... i Adding noise to the weight column can make the added noise approximate quantization noise.

[0086] Furthermore, the amplitude of the Gaussian noise added by the AI ​​model quantization device 200 can be determined based on the distribution of the quantized values ​​of the weights in the weight set 2, thereby enabling the injected Gaussian noise to better fit the quantized noise. Specifically, the AI ​​model quantization device 200 can determine the quantization parameter α corresponding to each weight column based on the distribution of the second values ​​of the weights in the weight set 2. i And based on the quantization parameter α i Determine the quantization scaling factor Δ corresponding to this weight column. i Therefore, based on this quantization scaling factor Δ iThe noise injected into the weight column is determined, which also determines the magnitude of the injected noise. Since Gaussian noise is injected into the weight columns (low sensitivity) in weight set 2, no Gaussian noise is injected into the weight columns (high sensitivity) in weight set 1. This helps reduce the magnitude of the Gaussian noise injected into the overall AI model (because the quantization noise from the higher-sensitivity weight columns is much greater than that from the lower-sensitivity weight columns). Consequently, quantizing only the weight columns in weight set 2 subsequently can also effectively reduce the overall quantization noise of the AI ​​model.

[0087] It is worth noting that the above description uses the example of AI model quantization device 200 adding Gaussian noise to the submatrix corresponding to weight set 2 on a column-by-column basis. In other embodiments, AI model quantization device 200 can also add Gaussian noise to the weights belonging to weight set 2 in the AI ​​model in a similar manner, with each weight as the granularity. Alternatively, AI model quantization device 200 can add other types of noise to the weights in a similar manner, without limitation.

[0088] S204: The AI ​​model quantization device 200 uses the dataset to fine-tune the AI ​​model after adding noise. During the fine-tuning of the AI ​​model, the values ​​of the weights belonging to weight set 1 in the AI ​​model are updated, while the values ​​of the weights belonging to weight set 2 in the AI ​​model are not updated.

[0089] The dataset used for fine-tuning the AI ​​model may be the same as or different from the dataset used to instruct the AI ​​model to run in step S201 above; there is no limitation on this.

[0090] In this embodiment, before quantizing the AI ​​model, the AI ​​model quantization device 200 can fine-tune the AI ​​model after adding noise using a dataset. The fine-tuning process can be performed on computing nodes 101 to 104. Furthermore, during the fine-tuning process, the AI ​​model quantization device 200 can freeze the weight values ​​in the weight set 2 after adding noise to the AI ​​model, adjusting only the weight values ​​in the weight set 1 within the AI ​​model. Since the noise injected into the AI ​​model simulates the noise generated by AI model quantization, the AI ​​model quantization device 200, by fine-tuning the weight values ​​in weight set 1, can minimize the impact of the injected noise (i.e., eliminate quantization noise) on the inference accuracy of the AI ​​model.

[0091] In one possible implementation, before fine-tuning the AI ​​model, the AI ​​model quantization device 200 can add noise to the weight values ​​in weight set 2 within the computation graph corresponding to the AI ​​model. Thus, during the fine-tuning phase of the AI ​​model, the AI ​​model quantization device 200 can perform calculations based on the injected noise computation graph and update the weight values ​​in weight set 1. Specifically, during the calculation of each network layer in the AI ​​model, such as... Figure 5 As shown, computation nodes 101 to 104 add the values ​​of the weights belonging to weight set 2 in the AI ​​model to the noise, and then concatenate this sum with the values ​​of the weights in weight set 1. In practical applications, when the channels of the weight columns in the weight matrix are reordered, the AI ​​model quantization device 200 can also concatenate the weights in weight set 1 and weight set 2 according to the aforementioned correspondence (i.e., the correspondence between the position of the sorted weight columns and their original positions in the weight matrix) to recover the correct weight matrix. Then, the AI ​​model quantization device 200 can perform matrix calculations based on the concatenated weight matrix and the input activation values, such as... Figure 5 As shown, calculations can be performed using general matrix to matrix multiplication (GEMM). During matrix calculation, the input activation value can be split into two parts based on the weight distribution. One part is multiplied with the weights belonging to weight set 1 in the weight matrix, and the other part is multiplied with the weights belonging to weight set 2 in the weight matrix. The results of the two parts are then concatenated to obtain the complete calculation result (i.e., the activation value) for the weight matrix. Similarly, for each network layer, computation nodes 101 to 104 can all use... Figure 5 The calculation is performed in the manner shown to complete the forward computation process of the AI ​​model. In the backward computation stage, the AI ​​model quantization device 200 can perform gradient updates based solely on the values ​​of the weights in the weight set 1.

[0092] S205: AI model quantization device 200 quantizes the values ​​of the weights belonging to weight set 2 in the fine-tuned AI model.

[0093] The AI ​​model quantization device 200 can quantize the AI ​​model based on a predetermined quantization rule. For example, the quantization rule can be to quantize the values ​​of the weights (weights belonging to the weight set 2) in the AI ​​model from high precision to a value with a specified number of bits. The specified number of bits can be, for example, 3 bits, 4 bits, or 8 bits.

[0094] Furthermore, the AI ​​model quantization device 200 can quantize not only some weights in the AI ​​model, but also activation values ​​in the AI ​​model, such as quantizing activation values ​​in the AI ​​model from 32 bits to 16 bits.

[0095] The quantization rules can be provided by the user, for example, by the user providing the quantization rules to the AI ​​model quantization device 200 through a client. Alternatively, the quantization rules can be pre-configured in the AI ​​model quantization device 200, so that for any AI model, the AI ​​model quantization device 200 can quantize the values ​​of some weights in the AI ​​model to a specified number of bits based on the quantization rules.

[0096] In practical applications, the AI ​​model quantization device 200 can utilize a quantization algorithm to perform a quantization process for the AI ​​model. This quantization algorithm is configured with specified quantization rules, or it can perform the quantization process based on configured quantization rules. For example, the quantization algorithm used may be, for instance, generative pre-trained transformer quantization (GPTQ) or QUIK quantization (an algorithm for quantizing user weights and activation values), etc., and there is no limitation on this.

[0097] Because the AI ​​model quantization device 200 minimizes the impact of quantization noise on the inference accuracy of the AI ​​model by fine-tuning it before quantization, the inference accuracy of the AI ​​model remains high even after quantizing weights with low quantization sensitivity. In other words, the inference accuracy of the quantized AI model is close to that of the unquantized model. Furthermore, during the fine-tuning phase, only a small amount of data is needed to update the values ​​of some weights in the AI ​​model (i.e., updating the values ​​of weights with high quantization sensitivity), reducing the storage and computational resources required for fine-tuning. Additionally, since the AI ​​model is fine-tuned before quantization and achieves a high level of inference accuracy, users no longer need to perform fine-tuning on the quantized AI model (customer model), thus reducing the demands on the user side and improving the user experience.

[0098] In actual testing scenarios, the AI ​​models can specifically be LLaMa2-7B and LLaMa2-13B. Each AI model can be tested using five datasets: Boolq, Hellaswag, Swag, Winogrande, and Xwinograd. Testing includes assessing the AI ​​model's inference accuracy before quantization, its inference accuracy after quantization using the QUIK quantization algorithm, and its accuracy using the aforementioned datasets. Figure 2 The inference accuracy achieved by quantizing the method of the illustrated embodiment is as follows: Figure 6 The test results are shown below. Figure 6 As shown, based on the above Figure 2 The method in the illustrated embodiment quantizes the AI ​​model, and on average, the inference accuracy of the AI ​​model can be reduced without loss (or even slightly increased). This ensures that the inference accuracy of the quantized AI model can still be maintained at a high level while successfully reducing the amount of data in the AI ​​model.

[0099] It is worth noting that the above Figure 2 The embodiments shown are for illustrative purposes only. In actual applications, the specific implementation can be different. Figure 2 Based on the illustrated embodiments, some steps may be added, deleted, or replaced. For example, the AI ​​model quantization device 200 may not execute step S201, etc., and there is no limitation in this regard.

[0100] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0101] The above combination Figures 1 to 6 The AI ​​model quantization method provided in the embodiments of this application will be introduced. Next, the structure of the AI ​​model quantization device and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0102] See Figure 7 The diagram shows a structural schematic of an AI model quantization device 700, which includes:

[0103] The partitioning module 701 is used to divide multiple weights in the AI ​​model into a first weight set and a second weight set. The first weight set includes important weights, and the important weights are more sensitive to quantization than the weights in the second weight set. Sensitivity is used to indicate the deviation of the activation value generated by the weight after quantization from the activation value generated before quantization.

[0104] Add module 702 to add noise to the values ​​of weights belonging to the second weight set in the AI ​​model;

[0105] The fine-tuning module 703 is used to fine-tune the AI ​​model after adding noise using the dataset. During the fine-tuning process, the values ​​of the weights belonging to the first weight set in the AI ​​model are updated, while the values ​​of the weights belonging to the second weight set in the AI ​​model are not updated.

[0106] The quantization module 704 is used to quantize the values ​​of the weights belonging to the second weight set in the fine-tuned AI model.

[0107] In one possible implementation, module 702 is added to add Gaussian noise to the values ​​of weights in the AI ​​model that belong to the second weight set.

[0108] In one possible implementation, module 702 is added for:

[0109] Predict the quantized values ​​of the weights in the second weight set;

[0110] Noise is added to the values ​​of the weights in the second weight set based on the distribution of their quantized values.

[0111] In one possible implementation, the second weight set includes multiple weight columns, and the adding module 702 is used for:

[0112] Based on the distribution of the weights in the second weight set after quantization, the quantization scaling factor for each weight column in the multiple weight columns is determined. The quantization scaling factor is used to determine the magnitude of the noise added to the weight column.

[0113] Noise is added to the weight values ​​in each weight column based on the quantization scaling factor corresponding to each weight column.

[0114] In one possible implementation, the partitioning module 701 is used for:

[0115] Evaluate the sensitivity of each of the multiple weights in an AI model to quantization;

[0116] Based on the sensitivity of each weight to quantization, multiple weights are divided into a first weight set and a second weight set.

[0117] In one possible implementation, the partitioning module 701 is used for:

[0118] Based on the first ratio and the sensitivity of each weight to quantization, the weight columns of the first part of the weight matrix corresponding to multiple weights are assigned to the first weight set, and the weight columns of the second part of the weight matrix are assigned to the second weight set. Each weight column in the first weight set has a weight whose sensitivity to quantization is higher than the sensitivity of all weights in each weight column in the second weight set to quantization. The ratio between the number of weight columns in the first weight set and the number of weight columns in the second weight set is the first ratio.

[0119] Alternatively, based on the second ratio and the sensitivity of each weight to quantization, multiple weights are divided into a first weight set and a second weight set, wherein the sensitivity of each weight in the first weight set to quantization is no less than the sensitivity of each weight in the second weight set to quantization, and the ratio between the number of weights in the first weight set and the number of weights in the second weight set is the second ratio.

[0120] Alternatively, weights with a sensitivity to quantization greater than the first threshold can be assigned to the first weight set, and weights with a sensitivity to quantization no greater than the first threshold can be assigned to the second weight set.

[0121] Alternatively, the first type of weight columns in the weight matrix corresponding to multiple weights can be assigned to the first weight set, and the second type of weight columns in the weight matrix can be assigned to the second weight set. In the first type of weight columns, there are weights whose sensitivity to quantization is greater than the second threshold, while in the second type of weight columns, there are no weights whose sensitivity to quantization is greater than the second threshold.

[0122] In one possible implementation, the partitioning module 701 is used for:

[0123] Obtain the first value of each weight before quantization and the input activation value from a set of weights;

[0124] Predict the second value of each weight after quantization;

[0125] Calculate the difference between the first and second values ​​of each weight;

[0126] Based on the difference and the input activation value, the sensitivity of each weight to quantization is calculated.

[0127] because Figure 7 The AI ​​model quantization device 700 shown corresponds to the above. Figure 2 The AI ​​model quantization device 200 in the illustrated embodiment, therefore Figure 7 For details on the implementation of the AI ​​model quantization device 700 and its technical effects, please refer to the above. Figure 2 The relevant details in the illustrated embodiments are described in detail here, and will not be repeated here.

[0128] Figure 8 This application provides a schematic diagram of the hardware structure of a computing device 800, which, for example, can implement the above-described... Figure 2 AI model quantization device 200, etc., in the illustrated embodiment.

[0129] like Figure 8 As shown, the computing device 800 includes a processor 801, a memory 802, and a communication interface 803. The processor 801, memory 802, and communication interface 803 communicate via a bus 804, or via wireless transmission or other means. The memory 802 stores instructions, and the processor 801 executes the instructions stored in the memory 802. Further, the computing device 800 may also include a memory unit 805, which is connected to the processor 801, the storage medium 802, and the communication interface 803 via the bus 804. The memory 802 stores program code, and the processor 801 can call the program code stored in the memory 802 to perform the following operations:

[0130] The AI ​​model is divided into a first set of weights and a second set of weights. The first set of weights includes important weights. The important weights are more sensitive to quantization than the weights in the second set of weights. The sensitivity is used to indicate the deviation of the activation value generated by the weight after quantization from the activation value generated before quantization.

[0131] Noise is added to the values ​​of the weights in the AI ​​model that belong to the second weight set;

[0132] The AI ​​model with added noise is fine-tuned using a dataset. During the fine-tuning process, the values ​​of the weights belonging to the first weight set in the AI ​​model are updated, while the values ​​of the weights belonging to the second weight set in the AI ​​model are not updated.

[0133] The values ​​of the weights belonging to the second weight set in the fine-tuned AI model are quantized.

[0134] It should be understood that in this embodiment, the processor 801 can be a CPU, but it can also be other general-purpose processors, deep learning processors (DPUs), GPUs, NPUs, TPUs, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc., or it can be a combination of the above-mentioned processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0135] The memory 802 may include read-only memory and random access memory, and provides instructions and data to the processor 801. The memory 802 may also include non-volatile random access memory.

[0136] The memory 802 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0137] The communication interface 803 is used to communicate with other devices connected to the computing device 800. The bus 804 may include a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, all buses are labeled as bus 804 in the figure.

[0138] It should be understood that the computing device 800 according to the embodiments of this application may correspond to the AI ​​model quantization device 700 in the embodiments of this application, and may correspond to the device executing the embodiments of this application. Figure 2 The method executed by the AI ​​model quantization device in the illustrated method, and the above and other operations and / or functions implemented by the computing device 800, are respectively for the purpose of implementing... Figure 2 The process of the corresponding methods in [the document] will not be elaborated here for the sake of brevity.

[0139] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned AI model quantization method.

[0140] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0141] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0142] The computer program product can be a software installation package. When any of the aforementioned AI model quantization methods is required, the computer program product can be downloaded and executed on a computing device.

[0143] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0144] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment. The terms “first,” “second,” etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate, and this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0145] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for quantifying artificial intelligence (AI) models, characterized in that, The method includes: The AI ​​model divides multiple weights into a first weight set and a second weight set. The first weight set includes important weights, and the important weights have a higher sensitivity to quantization than the weights in the second weight set. The sensitivity is used to indicate the deviation of the activation value generated by the weight after quantization from the activation value generated before quantization. Noise is added to the values ​​of the weights in the AI ​​model that belong to the second weight set; The AI ​​model with added noise is fine-tuned using a dataset. During the fine-tuning process, the values ​​of the weights belonging to the first weight set in the AI ​​model are updated, while the values ​​of the weights belonging to the second weight set in the AI ​​model are not updated. The values ​​of the weights belonging to the second weight set in the fine-tuned AI model are quantized.

2. The method according to claim 1, characterized in that, Adding noise to the values ​​of weights belonging to the second weight set in the AI ​​model includes: Gaussian noise is added to the values ​​of the weights in the AI ​​model that belong to the second weight set.

3. The method according to claim 1 or 2, characterized in that, Adding noise to the values ​​of weights belonging to the second weight set in the AI ​​model includes: Estimate the quantized values ​​of the weights in the second weight set; Based on the distribution of the weights in the second weight set after quantization, noise is added to the weight values ​​in the second weight set.

4. The method according to claim 3, characterized in that, The second weight set includes multiple weight columns. The step of adding noise to the weight values ​​in the second weight set based on the quantized value distribution of the weights in the second weight set includes: Based on the distribution of the weights in the second weight set after quantization, a quantization scaling factor is determined for each of the multiple weight columns. The quantization scaling factor is used to determine the magnitude of the noise added to the weight column. Based on the quantization scaling factor corresponding to each weight column, noise is added to the value of the weight in each weight column.

5. The method according to any one of claims 1 to 4, characterized in that, The step of dividing the multiple weights in the AI ​​model into a first weight set and a second weight set includes: Evaluate the sensitivity of each of the multiple weights in the AI ​​model to quantization; Based on the sensitivity of each weight to quantization, the multiple weights are divided into a first weight set and a second weight set.

6. The method according to claim 5, characterized in that, The step of dividing the multiple weights into a first weight set and a second weight set based on the sensitivity of each weight to quantization includes: Based on the first ratio and the sensitivity of each weight to quantization, the weight columns of the first part of the weight matrix corresponding to the multiple weights are assigned to the first weight set, and the weight columns of the second part of the weight matrix are assigned to the second weight set. Each weight column in the first weight set has a weight whose sensitivity to quantization is higher than the sensitivity of all weights in each weight column in the second weight set to quantization. The ratio between the number of weight columns in the first weight set and the number of weight columns in the second weight set is the first ratio. Alternatively, based on the second ratio and the sensitivity of each weight to quantization, the plurality of weights are divided into a first weight set and a second weight set, wherein the sensitivity of each weight in the first weight set to quantization is not lower than the sensitivity of each weight in the second weight set to quantization, and the ratio between the number of weights in the first weight set and the number of weights in the second weight set is the second ratio. Alternatively, the weights whose sensitivity to quantization is greater than the first threshold can be assigned to the first weight set, and the weights whose sensitivity to quantization is not greater than the first threshold can be assigned to the second weight set. Alternatively, the first type of weight column in the weight matrix corresponding to the multiple weights can be assigned to the first weight set, and the second type of weight column in the weight matrix can be assigned to the second weight set. In the first type of weight column, there are weights whose sensitivity to quantization is greater than a second threshold, and in the second type of weight column, there are no weights whose sensitivity to quantization is greater than the second threshold.

7. The method according to claim 6, characterized in that, The evaluation of the sensitivity of each of the multiple weights in the AI ​​model to quantization includes: Obtain the first value of each of the multiple weights before quantization and the input activation value; Estimate the second value of each weight after quantization; Calculate the difference between the first value and the second value of each weight; The sensitivity of each weight to quantization is calculated based on the difference and the input activation value.

8. An artificial intelligence (AI) model quantification device, characterized in that, The device includes: The partitioning module is used to divide multiple weights in the AI ​​model into a first weight set and a second weight set. The first weight set includes important weights, and the important weights have a higher sensitivity to quantization than the weights in the second weight set. The sensitivity is used to indicate the deviation of the activation value generated by the weight after quantization from the activation value generated before quantization. An addition module is used to add noise to the values ​​of weights in the AI ​​model that belong to the second weight set; The fine-tuning module is used to fine-tune the AI ​​model after adding noise using a dataset. During the fine-tuning of the AI ​​model, the values ​​of the weights belonging to the first weight set in the AI ​​model are updated, while the values ​​of the weights belonging to the second weight set in the AI ​​model are not updated. The quantization module is used to quantize the values ​​of the weights belonging to the second weight set in the fine-tuned AI model.

9. The apparatus according to claim 8, characterized in that, The adding module is used to add Gaussian noise to the values ​​of the weights in the AI ​​model that belong to the second weight set.

10. The apparatus according to claim 8 or 9, characterized in that, The added module is used for: Estimate the quantized values ​​of the weights in the second weight set; Based on the distribution of the weights in the second weight set after quantization, noise is added to the weight values ​​in the second weight set.

11. The apparatus according to claim 10, characterized in that, The second weight set includes multiple weight columns, and the adding module is used for: Based on the distribution of the weights in the second weight set after quantization, a quantization scaling factor is determined for each of the multiple weight columns. The quantization scaling factor is used to determine the magnitude of the noise added to the weight column. Based on the quantization scaling factor corresponding to each weight column, noise is added to the value of the weight in each weight column.

12. The apparatus according to any one of claims 8 to 11, characterized in that, The partitioning module is used for: Evaluate the sensitivity of each of the multiple weights in the AI ​​model to quantization; Based on the sensitivity of each weight to quantization, the multiple weights are divided into a first weight set and a second weight set.

13. The apparatus according to claim 12, characterized in that, The partitioning module is used for: Based on the first ratio and the sensitivity of each weight to quantization, the weight columns of the first part of the weight matrix corresponding to the multiple weights are assigned to the first weight set, and the weight columns of the second part of the weight matrix are assigned to the second weight set. Each weight column in the first weight set has a weight whose sensitivity to quantization is higher than the sensitivity of all weights in each weight column in the second weight set to quantization. The ratio between the number of weight columns in the first weight set and the number of weight columns in the second weight set is the first ratio. Alternatively, based on the second ratio and the sensitivity of each weight to quantization, the plurality of weights are divided into a first weight set and a second weight set, wherein the sensitivity of each weight in the first weight set to quantization is not lower than the sensitivity of each weight in the second weight set to quantization, and the ratio between the number of weights in the first weight set and the number of weights in the second weight set is the second ratio. Alternatively, the weights whose sensitivity to quantization is greater than the first threshold can be assigned to the first weight set, and the weights whose sensitivity to quantization is not greater than the first threshold can be assigned to the second weight set. Alternatively, the first type of weight column in the weight matrix corresponding to the multiple weights can be assigned to the first weight set, and the second type of weight column in the weight matrix can be assigned to the second weight set. In the first type of weight column, there are weights whose sensitivity to quantization is greater than a second threshold, and in the second type of weight column, there are no weights whose sensitivity to quantization is greater than the second threshold.

14. The apparatus according to claim 13, characterized in that, The partitioning module is used for: Obtain the first value of each of the multiple weights before quantization and the input activation value; Estimate the second value of each weight after quantization; Calculate the difference between the first value and the second value of each weight; The sensitivity of each weight to quantization is calculated based on the difference and the input activation value.

15. A computing device, characterized in that, The computing device includes a processor and memory; The memory is used to store instructions, and the processor executes the instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 7.

17. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 7.

Images