Data quantification method and related equipment

Abstract

The invention discloses a data quantification method, which comprises the steps of operating an artificial intelligence AI model, acquiring data generated in the operation process of the AI model, the data comprising a plurality of groups, determining the abnormal degree of the data in the first group in the plurality of groups, the abnormal degree indicating the difference degree of the size of the data in the first group, and determining the difference degree of the size of the data in the second group in the plurality of groups. And determining parameters for performing exception suppression on the data in the first group according to the exception degree, suppressing the data in the first group according to the parameters for exception suppression, and quantifying the suppressed data in the first group. According to the method, the abnormal degrees of different groups are determined respectively, and the appropriate abnormal suppression parameters are selected for the different groups based on the abnormal degrees to perform abnormal value suppression, so that adaptive suppression of data in the different groups is realized, the situation of insufficient suppression or excessive suppression can be avoided, and the quantization precision of the AI model is improved.

Description

[0001] This application is a divisional application. The original application has the application number 202410231737.8 and the original application date is February 29, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a data quantization method, a data quantization system, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Technology

[0003] With the continuous development of artificial intelligence (AI) technology, the parameter scale of AI models (or simply models) is also constantly increasing. For example, the parameter scale of large-scale language models (LLMs), which are currently attracting widespread attention, can reach billions, tens of billions, hundreds of billions, or even trillions. Compared with conventional models, large models (such as the LLMs mentioned above) have stronger generalization capabilities and better performance on various tasks, thus having broad application prospects.

[0004] While increasing the parameter size can improve the performance and generalization ability of AI models, it also poses challenges to their deployment. Currently, the industry mainly uses model quantization to compress AI models. By deploying compressed AI models, memory movement and computational load can be reduced during inference, thereby accelerating inference and lowering inference costs.

[0005] However, large models exhibit outliers in their activation distribution compared to conventional models. Activation distribution refers to the distribution of activation values ​​in a model. These activation values ​​can be the inputs to the AI ​​model or features from intermediate layers. Outliers are isolated points in the activation distribution, typically extreme values. Outliers can render conventional quantization algorithms ineffective. Forcing outliers to zero can render the AI ​​model's accuracy completely unusable; therefore, outliers are crucial to the capabilities of AI models. Providing a high-precision, highly usable model quantization solution has become a key focus in the industry. Summary of the Invention

[0006] This application provides a data quantization method that, for data from multiple groups generated during the operation of an AI model, determines the degree of anomaly in each group, determines parameters for anomaly suppression in the corresponding groups based on the anomaly degree, performs adaptive anomaly suppression on different groups based on these parameters, and then quantizes the suppressed data. This method achieves high-precision and highly available model quantization. This application also provides a data quantization system, computing device cluster, computer-readable storage medium, and computer program product corresponding to the above data quantization method.

[0007] Firstly, this application provides a data quantization method. This method can be executed by a data quantization system. The data quantization system can be a software system. This software system can be integrated with model compression tools and model deployment tools, for example, as a functional module of a model compression tool or model deployment tool, or integrated as a plug-in. The software system can also be a standalone software system. The software system can be provided to users as a software package, which users can deploy themselves in a local data center or private cloud. The software system can also be deployed in a public cloud and provided to users as a cloud service. The data quantization system can also be a hardware system, such as a cluster of computing devices with data quantization capabilities, which executes the data quantization method of this application during runtime.

[0008] Specifically, the data quantization system can run an AI model, acquire the data generated during the model's operation, which includes multiple groups, and then determine the degree of anomaly in the data of the first group. The degree of anomaly indicates the extent of difference in the magnitude of the data in the first group. Next, the data quantization system can determine parameters for anomaly suppression of the data in the first group based on the degree of anomaly, suppress the data in the first group according to the anomaly suppression parameters, and then quantize the suppressed data in the first group.

[0009] This method determines the degree of anomaly for different groups and selects appropriate anomaly suppression parameters for different groups based on the degree of anomaly to suppress outliers. This achieves adaptive suppression of data in different groups, thereby avoiding insufficient or excessive suppression and improving the quantization accuracy of AI models.

[0010] In some possible implementations, the data generated during operation consists of the activation values ​​of each layer generated by the AI ​​model in at least one inference process. Considering the randomness of the distribution of activation values ​​generated by each layer in a single inference process, this application can sample the activation values ​​generated by each layer in multiple inference processes of the AI ​​model. Statistical analysis based on the activation values ​​generated by each layer in multiple inference processes can yield more representative activation values. Consequently, the parameters determined for anomaly suppression of activation values ​​are more accurate, resulting in better suppression effects.

[0011] In some possible implementations, the data in the first group consists of activation values ​​generated by at least one layer in the AI ​​model during inference, or activation values ​​generated by different layers of the same set of input data during the AI ​​model's inference process. The same set of input data can be input data from at least one channel. For example, when the input data is an image, the data in the first group can be activation values ​​generated by different layers of the R-channel input data during the AI ​​model's inference process.

[0012] This method supports grouping at the layer or channel level to achieve adaptive suppression of data from different layers or different channels.

[0013] In some possible implementations, the parameter for anomaly suppression is a scaling factor. Accordingly, when the anomaly level is less than a first threshold, the data quantization system can determine the scaling factor of the data in the first group as a first value; when the anomaly level is greater than or equal to the first threshold, the data quantization system can determine the scaling factor of the activation values ​​in the first group based on the activation values ​​and weights in the first group.

[0014] This method can determine the scaling factor in different ways according to the degree of anomaly, thereby enabling adaptive suppression of different groups according to the degree of anomaly, avoiding insufficient or excessive suppression from affecting the quantization accuracy of the AI ​​model.

[0015] In some possible implementations, when the anomaly level is less than or equal to a second threshold, the data quantization system determines a scaling factor for the activation values ​​in the first group based on the ratio of the maximum activation value to the maximum weight in the first group. When the anomaly level is greater than the second threshold, a transfer coefficient is searched that makes the quantization error of the data in the first group satisfy a condition, and the scaling factor for the activation values ​​in the first group is determined based on the maximum activation value, the maximum weight, and the transfer coefficient in the first group.

[0016] In this method, the case where the anomaly level is greater than or equal to the first threshold can be further divided into: greater than or equal to the first threshold and less than or equal to the second threshold; and greater than the second threshold. When the anomaly level of the data in the first group is greater than or equal to the first threshold and less than or equal to the second threshold, it indicates that the anomaly level is in the middle range, and the data quantization system can directly balance the distribution of activation values ​​and weights to determine the scaling factor. When the anomaly level of the data in the first group is greater than the second threshold, it indicates that the anomaly level is relatively large, and the data quantization system can introduce a transition coefficient, also known as... A coefficient is used to determine the scaling factor. This allows for fine-grained anomaly suppression to meet business requirements.

[0017] In some possible implementations, the data quantification system can also sort multiple groups based on the degree of anomaly in the data within each group, and restore the data in the target group to its pre-suppression state based on the sorting results. The target group is either the group with anomaly levels greater than a third threshold, or the top n groups sorted by anomaly level from highest to lowest.

[0018] This method takes into account that some network layers have a very large data range. In order to maximize the quantization accuracy of the AI ​​model, it can identify groups with large data ranges and back off their data, such as activation values, to achieve lossless accuracy as much as possible.

[0019] In some possible implementations, when the anomaly level of the data in the first group is less than the fourth threshold, the data quantization system can discard a first proportion of outliers in the suppressed data of the first group and quantize based on the maximum and minimum values ​​of the remaining data; when the anomaly level of the data in the first group is greater than or equal to the fourth threshold, the data quantization system can quantize based on the maximum and minimum values ​​of the data in the suppressed data of the first group.

[0020] This method can adaptively select a quantization algorithm that matches the degree of anomaly of the data in the first group, thereby achieving a balance between the ability to resist outlier interference and accuracy.

[0021] In some possible implementations, the data quantization system can also acquire the random probability of the first group. When the random probability is greater than a second value, the data quantization system can quantize the data in the suppressed first group. Compared to no quantization, random quantization is more in line with real-world scenarios, and compared to full quantization, random quantization can also avoid overfitting. Therefore, random quantization has better generalization accuracy.

[0022] In some possible implementations, the data in the first group includes the activation values ​​and weights of the first group. The data quantization system can perform an exponential transformation on the suppressed weights in the first group to obtain the transformed weights, then quantize the transformed weights, and perform an inverse exponential transformation on the quantized weights.

[0023] This method introduces an exponential function to equivalently transform the quantization formula, thereby obtaining a non-uniform, adaptive quantization resolution. This allows for more user-friendly quantization of weights containing outliers. The non-uniform quantization in this application is an element-wise operation. Compared to other non-uniform quantization methods based on table lookup or shifting, this application's non-uniform quantization can be implemented without special hardware and at a lower cost.

[0024] In some possible implementations, the data quantization system can search for an exponent that satisfies the quantization error of the weights in the suppressed first group, and then perform an exponential transformation on the weights in the suppressed first group based on the exponent. This can minimize the error introduced by the exponential transformation.

[0025] Secondly, this application provides a data quantification system. The system includes:

[0026] An outlier suppression module is used to run an artificial intelligence (AI) model, acquire data generated during the AI ​​model's operation, the data includes multiple groups, determine the degree of anomaly of the data in the first group among the multiple groups, the degree of anomaly indicates the degree of difference in the size of the data in the first group, determine parameters for anomaly suppression of the data in the first group based on the degree of anomaly, and suppress the data in the first group based on the anomaly suppression parameters.

[0027] The quantization module is used to quantize the data in the suppressed first group.

[0028] In some possible implementations, the data generated during the operation is the activation value generated by the AI ​​model at each layer during at least one inference process.

[0029] In some possible implementations, the data in the first group are activation values ​​generated by at least one layer of the AI ​​model during inference, or activation values ​​generated by different layers of the same set of input data during the inference process of the AI ​​model.

[0030] In some possible implementations, the parameter for anomaly suppression is a scaling factor, and the outlier suppression module is specifically used for:

[0031] When the degree of anomaly is less than a first threshold, the scaling factor of the data in the first group is determined to be a first value;

[0032] When the degree of abnormality is greater than or equal to the first threshold, the scaling factor of the activation value in the first group is determined based on the activation value and weight in the first group.

[0033] In some possible implementations, the outlier suppression module is specifically used for:

[0034] When the degree of abnormality is less than or equal to the second threshold, the scaling factor of the activation value in the first group is determined according to the ratio of the maximum activation value to the maximum weight in the first group.

[0035] When the degree of anomaly is greater than the second threshold, a transfer coefficient is searched that makes the quantization error of the data in the first group meet the conditions, and a scaling factor for the activation value in the first group is determined based on the maximum activation value, the maximum weight and the transfer coefficient in the first group.

[0036] In some possible implementations, the system further includes:

[0037] The calibration module is used to sort the multiple groups according to the degree of anomaly of the data in each group, and restore the data in the target group of the multiple groups to the data before suppression according to the sorting result of the multiple groups. The target group is a group whose anomaly degree is greater than a third threshold, or a group whose anomaly degree is sorted into the top n from high to low.

[0038] In some possible implementations, the quantization module is specifically used for:

[0039] When the abnormality of the data in the first group is less than the fourth threshold, the first proportion of abnormal values ​​in the suppressed data in the first group are discarded, and quantization is performed based on the maximum and minimum values ​​of the remaining data.

[0040] When the abnormality of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the data in the suppressed first group.

[0041] In some possible implementations, the quantization module is specifically used for:

[0042] Obtain the random probability of the first group;

[0043] When the random probability is greater than the second value, the data in the suppressed first group is quantized.

[0044] In some possible implementations, the data in the first group includes the activation values ​​and weights of the first group, and the quantization module is specifically used for:

[0045] The weights in the first group after suppression are subjected to an exponential transformation to obtain the transformed weights.

[0046] The transformed weights are quantized, and the inverse exponential transformation of the quantized weights is performed.

[0047] In some possible implementations, the quantization module is specifically used for:

[0048] The search term is the exponent that makes the quantization error of the weights in the first group after suppression satisfy the condition.

[0049] Based on the index, the weights in the suppressed first group are subjected to an exponential transformation.

[0050] Thirdly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, which includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory to cause the computing device or the computing device cluster to perform the data quantization method as described in the first aspect or any implementation thereof.

[0051] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct a computing device or a cluster of computing devices to execute the data quantization method described in the first aspect or any implementation thereof.

[0052] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, causes the computing device or cluster of computing devices to perform the data quantization method described in the first aspect or any implementation thereof.

[0053] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0054] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0055] Figure 1 A schematic diagram illustrating the distribution of activation values ​​in different network layers of an AI model provided in this application;

[0056] Figure 2 A schematic diagram of the architecture of a data quantization system provided in this application;

[0057] Figure 3 A flowchart illustrating an outlier suppression method provided in this application;

[0058] Figure 4 A schematic diagram of a post-training quantization calibration process provided for this application;

[0059] Figure 5 A comparative schematic diagram of uniform quantization and non-uniform quantization provided for this application;

[0060] Figure 6 A flowchart of a data quantization method provided in this application;

[0061] Figure 7A schematic diagram of a quantization interface provided in this application;

[0062] Figure 8 A schematic diagram of the structure of a computing device provided in this application;

[0063] Figure 9 This application provides a schematic diagram of the structure of a computing device cluster;

[0064] Figure 10 This application provides a schematic diagram of another computing device cluster structure.

[0065] Figure 11 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0066] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0067] First, some technical terms involved in the embodiments of this application will be introduced.

[0068] An AI model is a model built based on AI technology. AI technology can include, but is not limited to, machine learning (ML) and deep learning (DL). Accordingly, an AI model can be a neural network model trained through machine learning or deep learning.

[0069] AI models can include multiple network layers, such as convolutional layers, pooling layers, and fully connected layers. The parameters used in calculations with the input of each network layer include weights, also known as weight values. The input of the AI ​​model and the features of intermediate layers constitute the activation values ​​of the AI ​​model. The features of intermediate layers can be those extracted by network layers between the input and output layers, such as convolutional layers and pooling layers. Activation values ​​or weights can include at least one channel, where the channel of the activation value or weight can be understood as its dimension. For example, when the AI ​​model is a neural network model for image processing, the input can be an image, which can include three dimensions: red, green, and blue. Based on this, the activation values ​​can include three channels. Similarly, the weights of convolutional layers in an AI model can include elements of the convolutional kernel. Weights can include input channels and output channels, where the input channels are usually the same as the output channels of the previous layer, and the number of output channels can be equal to the number of convolutional kernels.

[0070] Model quantization refers to the process of introducing quantization parameters, such as scaling factors and biases (also known as offsets), to convert floating-point values ​​in an AI model into fixed-point values, thereby achieving AI model compression and inference acceleration. The floating-point values ​​in the AI ​​model can include activation values ​​and weights; model quantization can include the quantization of activation values ​​and the quantization of weights.

[0071] Post-training quantization (PTQ) directly converts a pre-trained floating-point model into a fixed-point model. The floating-point model can be in single-precision (Full Precise Float 32, FP32) or half-precision (float, FP16) format, while the fixed-point model can perform integer calculations. Fixed-point models retain a specific number of integer and decimal places, while floating-point models retain a specific number of significant digits and exponents. This process eliminates the need to retrain the original model (e.g., the pre-trained floating-point model) and allows for calibration or optimization of the quantized parameters, ensuring that the accuracy loss after quantization meets requirements.

[0072] Model quantization can consist of two parts: quantization of the weights and quantization of the activations. In some examples, 8-bit quantization can be performed on both the weights and activations; this quantization is also called W8A8, thus achieving the equivalent performance of 32 bits. Here, W represents the weights and A represents the activations. Specifically, the basic operation in a neural network includes the convolution operation of the weights and activations, denoted as W... A, where convolution operations can include multiplication and addition operations. If one of the terms is quantized to {-1, 1}, then the multiplication and addition operation is simplified to addition and subtraction operations. If both terms are quantized to {-1, 1}, the multiplication and addition operation is simplified to bitwise operations, which is very friendly to hardware computation.

[0073] To deploy AI models, especially on resource-constrained edge devices, model quantization techniques such as PTQ are commonly used to compress the AI ​​models. However, as the parameter size of AI models increases, outliers begin to appear in the activation values, which can render conventional quantization algorithms ineffective. Related data shows that large models with more than 6.7 billion parameters (denoted as 6.7 billion, abbreviated as 6.7B) are beginning to exhibit outliers in their activation values. Although the proportion of outliers is only 1%, the range is quite wide, ranging from hundreds to thousands.

[0074] For ease of understanding, this application also provides an example. Figure 1The diagram illustrates the distribution of activation values ​​across network layers in a large model with a parameter size of 70B and more than 500 layers. The activation values ​​of some network layers are significantly larger than those of other network layers, resulting in a noticeable outlier phenomenon that is not conducive to quantization.

[0075] Forcibly zeroing out of even the 1% of outliers mentioned above can render the accuracy of an AI model completely unusable, highlighting the critical role of outliers in AI model capabilities. To address this, the industry has proposed schemes to suppress outliers. Currently, the mainstream outlier suppression scheme reduces outliers in activation values ​​through equivalent transformations. This scheme introduces an offset factor (denoted as shift) and a scaling factor (denoted as scale) to reduce outliers.

[0076] The specific steps are as follows: 1. Introduce an offset factor shift to the activation values ​​of each network layer, so that the activation values ​​of all network layers are migrated to the range centered on 0; 2. Calculate the scaling interval t that minimizes the quantization error through grid search, and derive the corresponding scaling factor scale based on the scaling interval t; 3. Offset and scale the activation values ​​of all network layers.

[0077] However, grid search-based scale determination tends to narrow the range of all activation values. To achieve equivalent transformation, an inverse transformation of the weights is usually required. For example, multiplying an activation value by the scaling factor requires dividing the weight by the scaling factor, which makes the weight distribution more rugged. This is not conducive to quantization of smaller models, nor to quantization of network layers with relatively flat activation value distributions in larger models. Furthermore, the amplitude range of outliers varies widely across different network layers. If a single suppression method is used, there may be insufficient suppression (suppressed outliers are still significant, causing the quantization algorithm to fail) or oversuppression (suppressed activation values ​​are easy to quantize, but the weights are significantly more difficult to quantize; this phenomenon is also called over-suppression). Both insufficient and oversuppression can lead to a decrease in the accuracy of the AI ​​model.

[0078] In view of this, this application provides a data quantization method. This method considers the wide range of data size distribution generated by AI models during operation. For example, during inference, the amplitude range of outliers in the activation values ​​of different network layers or channels is wide. Using a single suppression method may result in insufficient or excessive suppression. Therefore, a grouped adaptive outlier suppression strategy is proposed. Specifically, the data generated during AI model operation can include multiple groups. For example, the activation values ​​or weights of each network layer can form a group. Based on the degree of anomaly in the data within a group, parameters for anomaly suppression are determined. Then, the data in the first group is suppressed according to these parameters, and the suppressed data in the first group is quantized. This method achieves adaptive suppression of data in different groups by determining the degree of anomaly for each group and selecting appropriate anomaly suppression parameters based on the degree of anomaly. This avoids insufficient or excessive suppression and improves the quantization accuracy of the AI ​​model.

[0079] The data quantization method of this application can be applied to scenarios of AI model compression or deployment, especially large model compression or deployment. Specifically, large models can be deployed on resource-constrained devices such as edge devices or on the cloud for tasks such as dialogue and language understanding. By using the data quantization method of this application to quantify the data during the operation of the large model, such as activation values ​​generated during the inference process of the large model and the weights of the large model, deployment costs can be reduced, the computational load of the AI ​​model can be reduced, inference can be accelerated, and inference performance can be effectively improved.

[0080] This method can be executed by a data quantization system. The data quantization system can be a software system. This software system can be integrated with model compression tools and model deployment tools, for example, as a functional module of a model compression tool or model deployment tool, or integrated as a plug-in. The software system can also be a standalone software system. In this case, the software system can be provided to the user as a software package, which the user can deploy on a single computing device. The software system can also be deployed in a public cloud and provided to the user as a cloud service. When deployed in a public cloud, the software system can be deployed in a single virtual machine in the public cloud, or distributed across different virtual machines in the public cloud. The data quantization system can also be a hardware system, such as a cluster of computing devices with data quantization capabilities, which executes the data quantization method of this application when running.

[0081] To make the technical solution of this application clearer and easier to understand, the system architecture of the data quantization system of this application will be introduced below with reference to the accompanying drawings.

[0082] SeeFigure 2 The diagram illustrates the architecture of a data quantization system 200. This system 200 quantifies data generated during the operation of an AI model, such as quantifying activation values ​​and / or weights generated during AI model inference. The AI ​​model can be a fully trained model, such as a pre-trained model, which can be a large model. In some examples, the AI ​​model can also be a small model.

[0083] The data quantization system 200 may include an outlier suppression module 202 and a quantization module 204. Further, the data quantization system 200 may also include a calibration module 206. The calibration module 206 can perform adaptive PTQ calibration, and the quantization module 204 can quantize data such as activation values ​​or weights. Before quantization or calibration, the data quantization system 200 can first use the outlier suppression module 202 to perform an equivalent transformation on the AI ​​model to suppress outliers in the data. Then, the suppressed data is input to the calibration module 206 for activation value backoff, achieving adaptive calibration. The calibrated data can then be input to the quantization module 204 for quantization. For activation values, the quantization module 204 can use random quantization; for weights, the quantization module 204 can use low-bit non-uniform quantization.

[0084] The functions of the outlier suppression module 202, quantization module 204, and calibration module 206 are described in detail below.

[0085] The outlier suppression module 202 is used to run the AI ​​model, acquire the data generated during the AI ​​model's operation, which includes multiple groups, determine the degree of anomalousness of the data in the first group among the multiple groups, the degree of anomalousness indicates the length of the difference in the size of the data in the first group, and then determine the parameters for anomaly suppression of the data in the first group based on the degree of anomalousness, and suppress the data in the first group according to the parameters for anomaly suppression.

[0086] The data generated during the process can be the activation values ​​generated by each layer of the AI ​​model during at least one inference process. The data in the first group can be the data generated by at least one layer of the AI ​​model during inference, such as the activation values ​​generated by at least one layer of the AI ​​model during inference. In some examples, the data in the first group can be the activation values ​​generated by the same set of input data at different layers during the AI ​​model's inference process. The input data can include multiple channels; for example, an image may include R, G, and B channels. Data from different channels can generate activation values ​​at different layers during the AI ​​model's inference process, and the data in the first group can be the activation values ​​generated by data from at least one channel at different layers during the AI ​​model's inference process. For ease of description, the data in the first group will be referred to as the activation values ​​generated by at least one layer of the AI ​​model during inference.

[0087] Since the degree of anomaly in data can vary across different groups, using a single suppression method may result in insufficient or excessive suppression, which could reduce the accuracy of the AI ​​model. Therefore, the outlier suppression module 202 of this application can use different methods to determine the parameters for anomaly suppression for groups with different degrees of anomaly, thereby improving the outlier suppression effect.

[0088] The parameters for anomaly suppression can include a scaling factor (scale), and further, an offset factor (shift). An example of anomaly suppression for activation values ​​is given. Anomaly suppression can be divided into two steps: the first step is to introduce an offset factor (Shift) to offset the activation value, and the second step is to introduce a scaling factor (scale) to scale the activation value.

[0089] The activation value offset factor Shift can be the average of the maximum and minimum activation values ​​in the first group. The activation value group is formed using the activation values ​​of the i-th layer of the AI ​​model from the first group. As an example, the activation value offset factor can be determined using the following formula:

[0090]

[0091] in, Characterizes the activation value offset factor.

[0092] By offsetting according to the above-mentioned activation value offset factor, the activation values ​​in the activation value group can be migrated to a range with 0 as the symmetry center, thereby minimizing the numerical range of the activation value group.

[0093] For scaling factor This application considers that the degree of anomalousness of outliers in different groups (e.g., outliers in different network layers) may vary, and introduces a factor to measure the degree of anomalousness, also called the degree of anomalousness, denoted as . The outlier suppression module 202 can determine the scaling factor of the data in the first group based on the degree of anomaly in the data in the first group.

[0094] The anomaly level indicates the degree of variation in the magnitude of the data within the first group. In some examples, the anomaly level can be the ratio of the largest absolute value of the activation values ​​in the activation value group to the standard deviation. Specifically, it can be determined using the following formula:

[0095]

[0096] in, Indicates the group of activation values The degree of abnormality, Representing absolute value, Characterizes the standard deviation. The larger the value, the more active the value group. The higher the degree of abnormality.

[0097] In its implementation, the outlier suppression module 202 can model the scaling factor and the degree of anomaly to obtain a mapping relationship between the scaling factor and the degree of anomaly. This mapping relationship can be represented by a function. In some examples, this function can be a piecewise function. When the range is different, the scaling factor can be calculated in different ways. To adapt to activation value quantization under different activation distributions.

[0098] Specifically, the outlier suppression module 202 is used to suppress outlier values ​​in the first group (such as the activation value group). When the anomaly level of the activation values ​​in the first group is less than a first threshold, the scaling factor of the data in the first group is determined to be a first value. When the anomaly level of the first group is greater than or equal to the first threshold, the scaling factor of the activation values ​​in the first group is determined based on the activation values ​​and weights in the first group. The first value can be a fixed value, such as 1, or another fixed value close to 1. The first threshold can be set empirically, and this embodiment does not impose any restrictions on it.

[0099] Furthermore, the case where the anomaly level is greater than or equal to the first threshold can be further divided into the following situations: ① greater than or equal to the first threshold and less than or equal to the second threshold; ② greater than the second threshold. The outlier suppression module 202 is used to determine the scaling factor of the activation value in the first group based on the ratio of the maximum activation value to the maximum weight in the first group when the anomaly level of the data in the first group is greater than or equal to the first threshold and less than or equal to the second threshold. When the anomaly level of the data in the first group is greater than the second threshold, it searches for a transfer coefficient that makes the quantization error of the data in the first group satisfy the condition. Based on the maximum activation value, the maximum weight, and the transfer coefficient in the first group, it determines the scaling factor of the activation value in the first group.

[0100] The following will illustrate this with specific examples. For example... Figure 3 As shown, the outlier suppression module 202, according to Scope, will It is divided into three tiers, and each tier offers corresponding... Calculation method. First tier, If the range is minimized, then it can be assumed that there are no outliers or that the impact of outliers is minor (negligible), so outlier suppression is not necessary. It can be set to 1; the second level. With the range centered, in this case, the distribution of activation values ​​and weights can be directly balanced and calculated. Third tier: The range is the largest, and the degree of anomaly is the greatest, so a transition coefficient can be introduced, also known as... coefficients, by setting The coefficients can be set to a range, for example, 0-1 (including endpoints), and then a fine-grained search method can be used to find the optimal solution. To ensure the quantization error is minimized, then based on the solution obtained... Determining the coefficients, activation values ​​in the first group, and weights in the first group. The quantization error can be the mean-square error (MSE).

[0101] The above three levels Range corresponding It can be determined using the following formula:

[0102]

[0103] in, A scaling factor representing the activation values ​​in the first group, for example, is the activation value group. scaling factor, Representing the weights in the first group, for example, a weight group. Characterizes the transfer coefficient. Characterizing the activation value group The degree of abnormality, The first threshold, This is the second threshold.

[0104] The outlier suppression module 202 can scale the data (e.g., activation values) in the first group according to a scaling factor to obtain a suppressed first group. Specifically, after suppressing the activation values ​​in the first group using the scaling factor, the outlier suppression module 202 can further divide the weights in the first group by the scaling factor to achieve an equivalent transformation. This allows for outlier suppression based on data distribution tiers during AI model operation, avoiding insufficient or excessive suppression due to a single suppression method, thus contributing to improved quantization accuracy.

[0105] The quantization module 204 is used to quantize the data in the suppressed first group. The granularity of data quantization can be the same as that of outlier suppression. For example, the granularity of activation value quantization can be group-by-group quantization, specifically quantization is performed on a group-by-group basis, with a group of activation values ​​sharing the quantization parameters.

[0106] In some possible implementations, after suppressing outliers in the data generated during the operation of AI models (such as large models), the influence of outliers may not be completely eliminated. Based on this, in addition to the outlier suppression schemes mentioned above, this application also provides an accuracy preservation strategy during the calibration phase, including but not limited to activation value rollback strategies.

[0107] Specifically, the first group includes activation values ​​in at least one network layer of the AI ​​model. Considering that the activation value range of some network layers is very large, in order to maximize the quantization accuracy of the AI ​​model, this application introduces activation value backoff. Activation value backoff refers to reversing the processing of activation values ​​that have undergone outlier suppression, restoring them to their activation values ​​before outlier suppression.

[0108] Specifically, see Figure 4 The diagram illustrates a data calibration and quantization process. Calibration module 206 sorts multiple groups based on the degree of anomaly in each group, for example, from highest to lowest, obtaining a sorting result. Then, based on this sorting result, it restores the data in a target group to its pre-suppression state, for example, by reverting the activation values ​​in the target group back to their pre-suppression values. The target group can be the top n groups sorted by anomaly degree from highest to lowest, or groups with anomalies exceeding a third threshold. Related research indicates that n values ​​between 1 and 5 can achieve lossless accuracy. The third threshold can be set empirically. For example, calibration module 206 can revert the activation values ​​of the top 5 network layers in terms of anomaly degree, restoring them to their pre-suppression values. This combines the characteristics of large model quantization, achieving layer-level quantization sensitivity in O(n) computational complexity, which is more efficient than traditional mixed-precision algorithms.

[0109] Furthermore, this application supports multiple quantization algorithms when quantizing the data in the suppressed first group. The quantization module 204 can select the appropriate algorithm for quantization based on the degree of anomalousness of the data in the first group. Specifically, when the degree of anomalousness of the data in the first group is less than a fourth threshold, the quantization module 204 discards a first proportion of outliers in the suppressed first group and quantizes based on the maximum and minimum values ​​of the remaining data; when the degree of anomalousness of the data in the first group is greater than or equal to the fourth threshold, the quantization module 204 quantizes based on the maximum and minimum values ​​of the data in the suppressed first group.

[0110] Let's take activation value quantization as an example. Activation value quantization algorithms can include various types, such as mini-maximum quantization and histogram quantization. Mini-maximum quantization calculates the minimum and maximum values ​​of multiple data sets and directly uses them as the floating-point range. Even with large outliers, it will still treat outliers as minimums or maxima. Histogram quantization models multiple data sets using histograms, discarding outliers in the first proportion and using the median distribution as minimums and maxima. It has strong resistance to outlier interference. When outliers are small, the range calculated based on the modeling method is more accurate than mini-maximum quantization; however, when outliers are large, histogram-based quantization directly eliminates outliers, leading to a significant decrease in accuracy.

[0111] Therefore, this application supports adaptive selection of activation value quantization algorithms based on data distribution. The quantization module 204 employs two algorithms for statistically analyzing activation value ranges: minimum-maximum and histogram-based algorithms. During activation value quantization, it adaptively selects the optimal algorithm based on the data distribution. When the value is less than the fourth threshold, the quantization module 204 can select histogram quantization to quantize the activation values ​​in the suppressed first group. When the value is greater than or equal to the fourth threshold, the quantization module 204 can select minimax quantization to quantize the activation values ​​in the suppressed first group.

[0112] Since different activation value quantization algorithms are suitable for activation values ​​with different data distributions, adaptively selecting the optimal algorithm based on the characteristics of the activation distribution of the AI ​​model can improve quantization accuracy.

[0113] During the layer-by-layer quantization calibration process, the data (e.g., activation values) from all preceding layers can be used to determine whether to quantize using random probabilities. Specifically, the quantization module 204 can obtain the random probability of the first group. When the random probability is greater than a second value, the data in the suppressed first group is quantized. Compared to no quantization, random quantization is more consistent with real-world scenarios, and compared to full quantization, random quantization also avoids overfitting. Therefore, random quantization has better generalization accuracy.

[0114] The activation value quantization process has been explained in detail above. The quantization process can also include weight quantization. The weight quantization process will be explained in detail below.

[0115] The quantization module 204 is also used to quantize the weights of the AI ​​model. The weights of AI models, such as those of large models, are unevenly distributed and contain a certain proportion of outliers; the largest weight can be more than 100 times larger than a normal weight, posing a challenge to low-bit quantization below 8 bits. Figure 5As shown, ordinary uniform quantization lacks adaptive discrimination capability in its quantization resolution, which leads to a significant waste of precision bits. Non-uniform quantization, on the other hand, can significantly improve accuracy. Based on this, this application also provides an outlier adaptive method to achieve high-precision low-bit non-uniform quantization.

[0116] In this application, the core idea of ​​non-uniform quantization is to introduce an exponential function to equivalently transform the quantization formula, thereby obtaining a non-uniform, adaptive quantization resolution. This allows for more user-friendly quantization of weights containing outliers. Specifically, this application uses a method that minimizes the error to solve for the exponent of the exponential transformation of each weight. The non-uniform quantization in this application is an element-wise operation. Compared to other non-uniform quantization methods based on table lookup or shifting, this application's non-uniform quantization can be implemented without special hardware and at a lower cost.

[0117] The non-uniform transformation of this application will be described in detail below.

[0118] In specific implementation, the quantization module 204 performs an exponential transformation on the weights in the suppressed first group to obtain the transformed weights. Then, it quantizes the transformed weights and performs an inverse exponential transformation on the quantized weights to achieve equivalence before and after the transformation. The weights in the first group can include weights in at least one channel of the AI ​​model, or the first group can include weights in at least one network layer of the AI ​​model. For example, the weights of each network layer of the AI ​​model can form a weight group. Alternatively, the weights in each channel of the AI ​​model can form a weight group. In some cases, weights from multiple network layers or channels with the same or similar distribution can also form a weight group.

[0119] The example of exponential transformation of the weights in the first group is illustrated below. Specifically, the quantization module 204 is used to search for an exponent that makes the quantization error of the weights in the suppressed first group meet the conditions, and then performs an exponential transformation on the weights in the first group based on the exponent.

[0120] The quantization module 204 can introduce formula (4) to perform an exponential transformation on the weights, making the weight distribution non-uniform. Then, formula (5) is used to quantize the transformed weights f(W), and the exponential transformation is reversed after quantization to ensure equivalence before and after the transformation. The exponent α used for the transformation can be obtained through a search. The range of α is between 0 and 1. The final selected α is the α that minimizes the weight quantization error, as shown in formula (6).

[0121]

[0122] In the above formula, Q represents quantization. To revert to the quantized weight values ​​after the exponential transformation. This represents the weight values ​​of the l-th layer. Let be the order of the norm, for example A value of 2 indicates the L2 norm. This indicates the weight quantization error.

[0123] Considering that a small number of outliers exist in the weights of AI models, non-uniform quantization can avoid wasting precision bits, improve accuracy, and achieve low-bit quantization. Furthermore, this application employs exponential transformation to achieve non-uniform quantization, thereby improving quantization efficiency.

[0124] Based on the aforementioned data quantization system 200, this application also provides a data quantization method. The data quantization method of this application will be described below with reference to embodiments.

[0125] See Figure 6 The flowchart shown illustrates a data quantization method, which includes the following steps:

[0126] S602, Data Quantization System 200 runs the AI ​​model and acquires the data generated during the AI ​​model's operation.

[0127] The data generated during the operation of an AI model can be data generated during the AI ​​model's inference process, such as the activation values ​​of the AI ​​model. In some examples, the data generated during the operation of the AI ​​model may also include the weights of the AI ​​model, such as the weights that are iteratively updated during the training process. Activation values ​​and weights are usually floating-point data, which can be quantized to improve inference performance.

[0128] To improve quantization results, the data quantization system 200 can perform multiple inference operations on the AI ​​model and perform statistical analysis on the data generated from these operations to obtain the data generated during the AI ​​model's operation. Taking activation values ​​as an example, the data quantization system 200 can perform m inference operations on the AI ​​model to obtain the activation values ​​generated from these m inference operations. Then, the data quantization system 200 can calculate the average value of these m activation values, or determine the maximum and minimum values. The data quantization system 200 can then use the average, maximum, or minimum value of the activation values ​​generated from these m inference operations as the data generated during the AI ​​model's operation to determine the quantization parameters.

[0129] The data generated during the operation of an AI model can include multiple groups. Taking the first group as an example, the data in the first group can be activation values ​​generated by at least one layer in the AI ​​model during inference. Alternatively, the data in the first group can be activation values ​​generated by different layers of the same set of input data during the AI ​​model's inference process; for example, activation values ​​generated by different layers of input data from a single channel during the AI ​​model's inference process. For instance, in an image processing scenario, the input image of the AI ​​model can include three channels: R, G, and B. The first group could be activation values ​​generated by different layers of the R channel input image during the AI ​​model's inference process. Similarly, the data in the first group can be weights of at least one layer in the AI ​​model, or weights used to process input data from at least one channel.

[0130] Specifically, the data quantization system 200 can provide an interface through which the AI ​​model to be quantized can be obtained. This interface can be an application programming interface (API) or a user interface, such as a graphical user interface (GUI) or a command user interface (CUI).

[0131] In some possible implementations, the model training tool can call the API provided by the data quantization system 200, providing the AI ​​model to be quantized. For example, the API interface parameters may include the path or storage address of the AI ​​model to be quantized. The data quantization system 200 can then retrieve the parameters of the AI ​​model from the corresponding path or storage address, such as the AI ​​model's structure and weights. The data quantization system 200 inputs the input data into the AI ​​model for layer-by-layer inference, obtaining activation values.

[0132] In some other possible implementations, the data quantification system 200 can present a quantification interface to the user. For example... Figure 7As shown, the quantization interface 700 can be a GUI, containing a path input box 702 and an upload control 704. Users can enter the path of the AI ​​model to be quantized in the path input box and then click the upload control 704 to trigger an upload operation. Correspondingly, the data quantization system 200 can respond to the user-triggered upload operation and obtain the AI ​​model to be quantized. In some examples, the quantization interface 700 may also include a browsing control 706, allowing users to select the AI ​​model to be quantized and input its address into the path input box 702, eliminating the need for manual input and improving convenience. Furthermore, the data quantization system 200 can input the input data into the AI ​​model, run the AI ​​model for layer-by-layer inference, and obtain the activation values ​​for each layer.

[0133] S604, Data Quantization System 200 determines the degree of anomaly in the data of the first group among multiple groups.

[0134] AI anomaly level indicates the degree of difference in the size of data within a first group. This difference can be absolute, such as the difference between the maximum and minimum values ​​in the first group. Alternatively, it can be relative, such as the ratio of the maximum absolute value to the standard deviation in the first group. Taking activation values ​​as an example, the anomaly level of activation values ​​in the first group can be the ratio of the maximum absolute value to the standard deviation. Based on this, the data quantification system 200 can first perform statistical analysis on the activation values ​​in the first group to determine the maximum absolute value and the standard deviation of the activation values, and then determine the anomaly level of the activation values ​​in the first group based on the aforementioned maximum absolute value and standard deviation.

[0135] It should be noted that using the ratio of the maximum absolute value to the standard deviation as the degree of anomaly is only one exemplary implementation of this application. In other possible implementations of this application, the degree of anomaly can also be other indicators and determined in other ways. For example, the degree of anomaly can also be the ratio of the maximum absolute value to the variance. As another example, the degree of anomaly can also be the sum of the ratio of the maximum absolute value to the standard deviation and a set value.

[0136] S606, Data Quantization System 200 determines the parameters for anomaly suppression of data in the first group based on the degree of anomaly.

[0137] Considering that the degree of anomaly in data may vary in different groups, if the same anomaly suppression parameters are used for scaling, such as using the same scaling factor to uniformly scale the activation values ​​in multiple groups, it may lead to insufficient or excessive suppression. Therefore, the data quantization system 200 can determine the scaling factor of the data in the first group based on the degree of anomaly in the data in the first group.

[0138] Specifically, the data quantization system 200 can model the scaling factor and the degree of anomaly to obtain a mapping relationship between the scaling factor and the degree of anomaly. Accordingly, when determining the scaling factor, the data quantization system 200 can determine the scaling factor for the data in the first group based on the degree of anomaly in the data, through the aforementioned mapping relationship between the scaling factor and the degree of anomaly.

[0139] The mapping relationship between the scaling factor and the degree of anomaly can be represented by a function, the output of which is the scaling factor. The function can be a piecewise function related to the degree of anomaly. When the range of anomaly degree is different, the scaling factor Scale can be calculated in different ways to adapt to data quantization under different data distributions.

[0140] The degree of anomaly can be categorized based on its relationship to a threshold. In some possible implementations, the degree of anomaly can be divided into the following categories based on its relationship to a first threshold: less than the first threshold; greater than or equal to the first threshold. When the degree of anomaly in the data of the first group is less than the first threshold, it indicates that the degree of anomaly in the first group is small, and it can be considered that there are no outliers or that the impact of outliers is small. The data quantization system 200 can determine the scaling factor of the data in the first group as a first value. The first value can be a fixed value, for example, the first value can be 1, thus ensuring that the data in the first group is not suppressed. When the degree of anomaly in the data of the first group is greater than or equal to the first threshold, the data quantization system 200 can determine the scaling factor of the activation values ​​in the first group based on the activation values ​​and weights in the first group. The weights include the weights in the network layer or channel to which the activation values ​​in the first group belong. For example, if the first group includes activation values ​​in the i-th network layer, then the weights include the weights in the i-th network layer.

[0141] Furthermore, the cases where the anomaly level is greater than or equal to the first threshold can be further divided into: greater than or equal to the first threshold and less than or equal to the second threshold; and greater than the second threshold. When the anomaly level of the data in the first group is greater than or equal to the first threshold and less than or equal to the second threshold, it indicates that the anomaly level is in the middle range, and the data quantization system 200 can directly balance the distribution of activation values ​​and weights to determine the scaling factor. Specifically, the data quantization system 200 can determine the scaling factor of the activation values ​​in the first group based on the ratio of the maximum activation value to the maximum weight in the first group. When the anomaly level of the data in the first group is greater than the second threshold, it indicates that the anomaly level is relatively large, and the data quantization system 200 can introduce a transfer coefficient, also known as... The scaling factor is determined by the coefficients. Specifically, the data quantization system 200 can search for transfer coefficients that satisfy the condition for the quantization error of the data in the first group, wherein the data quantization system 200 can set... The coefficients are set to a range, for example, 0-1 (including endpoints), and then a fine-grained search method is used to solve for the optimal value. The coefficients are used to ensure that the quantization error of the activation value is minimized. Then, based on the maximum activation value, the maximum weight, and the obtained values ​​from the first group, the calculation is performed. The coefficients determine the scaling factor for the activation values ​​in the first group.

[0142] Among them, the data quantification system 200 is based on The specific implementation of the scaling factor determined by the coefficient, maximum activation value, and maximum weight can be found in the relevant content of formula (3), and will not be repeated here.

[0143] In addition, the parameters for anomaly suppression can also include a shift factor. Taking activation values ​​as an example, the shift factor can be the average of the activation values ​​in the first group, such as the arithmetic mean of the maximum and minimum activation values. For specific implementation details, please refer to the relevant content description in formula (1).

[0144] S608, Data Quantization System 200 suppresses the data in the first group according to the parameters of anomaly suppression.

[0145] Data within the same group can share anomaly suppression parameters. For example, activation values ​​within the same group can share scaling factors. The data quantization system 200 can use the scaling factor of the data in the first group to scale the data (e.g., activation values) to obtain a suppressed first group. Furthermore, to achieve an equivalent transformation, the data quantization system 200 can perform the inverse scaling process on the corresponding weights. For example, the data quantization system 200 can scale the activation values ​​and then divide the weights by the scaling factor to achieve an equivalent transformation.

[0146] Furthermore, the scaling factors for different groups can be represented by vectors, where each element of the vector represents the scaling factor for a group. Accordingly, the data quantization system 200 can scale the data in different groups using matrix multiplication operations, thus improving efficiency.

[0147] In some possible implementations, the data quantization system 200 can also determine an offset factor based on the maximum and minimum activation values ​​in the group, and offset the activation values ​​according to this offset factor, thereby migrating the activation values ​​in the group to a range centered on 0, minimizing the numerical range of the activation values. The data quantization system 200 then performs the aforementioned scaling process on the offset activation values ​​to suppress outliers.

[0148] S610, the data quantization system 200 quantizes the data in the suppressed first group.

[0149] Specifically, the data quantization system 200 can determine the granularity of data quantization and quantize the suppressed data in the first group according to that granularity. In some examples, the granularity of data quantization is the same as the granularity of outlier suppression. For example, the granularity of data quantization can be group-by-group quantization, specifically quantizing on a per-activation-value-group basis, where an activation-value-group shares quantization parameters.

[0150] Considering that outlier suppression cannot completely eliminate the influence of outliers, the data quantization system 200 can also ensure the quantization accuracy of the AI ​​model during the calibration phase by backtracking data, for example, by backtracking the activation values ​​of some groups to the activation values ​​before outlier suppression.

[0151] Specifically, the data in the first group can be activation values ​​generated by at least one layer of the AI ​​model during inference. Considering that the activation value range of some network layers is very large, this application introduces activation value backoff to maximize the quantization accuracy of the AI ​​model. Specifically, the data quantization system 200 can sort multiple groups according to the degree of anomaly of the data in each group, for example, sorting them in descending order of anomaly degree. Based on the sorting results of multiple groups, the data in the target group among the multiple groups is restored to the data before suppression, for example, restored to the activation values ​​before suppression. The target group can be the top n groups in the sorting results, or the groups with anomaly degree greater than a third threshold. n and the third threshold can be set empirically, for example, n can be 1~5 to achieve lossless accuracy. In some examples, the data quantization system 200 can save the activation values ​​before suppression, and can read the activation values ​​before suppression when backoff is needed. In other examples, the data quantization system 200 can perform inverse processing on the activation values ​​for outlier suppression. For example, when performing activation value rollback on the first activation value group, if the scaling factor of the activation values ​​in the first group is 0.2, the suppressed activation values ​​in the first group can be divided by the above scaling factor (equivalent to multiplying by the reciprocal of the scaling factor, such as multiplying by 5) to achieve activation value rollback.

[0152] Furthermore, this application supports multiple quantization algorithms, including but not limited to mini-maximum quantization and histogram quantization. Different quantization algorithms are suitable for data with different distributions; therefore, the data quantization system 200 can adaptively select a quantization algorithm based on the characteristics of the data distribution in a group. Specifically, when the anomaly level of the data in the first group is less than a fourth threshold, the data quantization system 200 can perform histogram quantization on the suppressed data in the first group. Specifically, the data quantization system 200 can discard a first proportion of outliers in the suppressed data in the first group and quantize based on the maximum and minimum values ​​of the remaining data. When the anomaly level of the data in the first group is greater than or equal to the fourth threshold, the data quantization system 200 can perform mini-maximum quantization on the suppressed data in the first group. Specifically, the data quantization system 200 can quantize based on the maximum and minimum values ​​of the suppressed data in the first group.

[0153] Let's continue with the example of activation values ​​in the first group. Specifically, when the anomaly level of the activation values ​​in the first group is less than the fourth threshold, it indicates that the anomaly level of the activation values ​​in the first group is relatively small, and the data quantization system 200 can perform histogram quantization on the suppressed activation values ​​in the first group. When the anomaly level of the activation values ​​in the first group is greater than or equal to the fourth threshold, it indicates that the anomaly level of the activation values ​​in the first group is relatively large, and the data quantization system 200 can perform minimax quantization on the suppressed activation values ​​in the first group.

[0154] During the layer-by-layer quantization calibration process, the activation values ​​of all preceding layers for each layer can be used to determine whether to quantize using random probabilities. Specifically, for the first group, the data quantization system 200 can obtain the random probability of the first group. When the random probability is greater than a second value, the data in the suppressed first group can be quantized. Compared to no quantization, random quantization is more in line with real-world scenarios, and compared to full quantization, random quantization can also avoid overfitting. Therefore, random quantization has better generalization accuracy.

[0155] The activation value quantization process has been explained above. The data quantization system 200 can also quantify weights. The weight quantization process will be explained in detail below.

[0156] Considering that the weight distribution of AI models is uneven and there is a certain proportion of outliers, if uniform quantization is used, a large number of precision bits will be wasted due to the lack of adaptive partitioning capability of quantization resolution. Therefore, the data quantization system 200 can adopt a non-uniform quantization mechanism to quantize the weights.

[0157] Non-uniform quantization can be achieved by introducing an exponential function to equivalently transform the quantization formula, resulting in a non-uniform, adaptive quantization resolution. This allows for more effective quantization of weights containing outliers. Specifically, the data quantization system 200 can perform an exponential transformation on the weights in the suppressed first group to obtain the transformed weights. Then, the data quantization system 200 can quantize the transformed weights and perform an inverse exponential transformation on the quantized weights to ensure equivalent transformation.

[0158] When performing exponential transformation, the data quantization system 200 can first determine the exponent. Taking the first group as an example, the data quantization system 200 can search for an exponent that satisfies the conditions for weight quantization error, and then perform exponential transformation on the weights in the first group according to the exponent.

[0159] The process of determining the index, performing index transformation based on the index, and quantizing based on the index transformation results can be found in [reference needed]. Figure 2 The illustrated embodiment is, for example, Figure 2 The relevant content of formulas (4) to (6) in the illustrated embodiment will not be repeated here.

[0160] As described above, the data quantization method of this application considers the wide range of outlier amplitudes in different network layers or channels of the AI ​​model. Using a single suppression method may result in insufficient or excessive suppression. Therefore, this application proposes outlier suppression based on the data distribution of different groups. For each group, the degree of anomaly is determined according to its data distribution, and the parameters for anomaly suppression within that group are determined based on the degree of anomaly. This avoids insufficient or excessive suppression. Adaptive suppression for different groups improves the quantization accuracy of the AI ​​model. Furthermore, during the quantization of activation values, activation value backoff, adaptive calibration, or random quantization can be used to ensure the quantization accuracy of the AI ​​model and improve generalization ability. In addition, for weight quantization, this application introduces exponential transformation to achieve high-precision, low-bit non-uniform quantization, reducing the waste of precision bits.

[0161] Based on the aforementioned data quantization method, this application also provides a data quantization system 200. For example... Figure 2 As shown, the data quantization system 200 includes:

[0162] An outlier suppression module 202 is used to run an AI model, acquire data generated during the operation of the AI ​​model, the data includes multiple groups, determine the degree of anomaly of the data in the first group among the multiple groups, the degree of anomaly indicates the degree of difference in the size of the data in the first group, determine the parameters for anomaly suppression of the data in the first group based on the degree of anomaly, and suppress the data in the first group based on the parameters for anomaly suppression.

[0163] Quantization module 204 is used to quantize the data in the suppressed first group.

[0164] The above-mentioned outlier suppression module 202 is used to implement Figure 6 In this embodiment, the content related to S602 to S608 is described, and the quantization module 204 is used to implement the content related to S610. The outlier suppression module 202 and the quantization module 204 can be implemented in hardware or in software.

[0165] When implemented in software, the outlier suppression module 202 and quantization module 204 can be applications running on computing devices, such as computing engines. These applications can also be virtualized and provided to users through virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can be services that use virtualization technology to create virtual machine (VM) resource pools on multiple physical hosts to provide VMs for users to use on demand. BMS services are services that use virtualization technology to create BMS resource pools on multiple physical hosts to provide BMS for users to use on demand. Container services are services that use virtualization technology to create container resource pools on multiple physical hosts to provide containers for users to use on demand. A VM is a simulated virtual computer, that is, a logical computer. A BMS is a scalable, high-performance computing service with computing performance indistinguishable from traditional physical machines and features secure physical isolation. A container is a kernel virtualization technology that can provide lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely specific examples. In practical applications, virtualization services can also include other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0166] When implemented in hardware, the outlier suppression module 202 and the quantization module 204 may include at least one computing device, such as a server. Alternatively, the outlier suppression module 202 and the quantization module 204 may also be devices implemented using application-specific integrated circuits (ASICs) or programmable logic devices (PLDs). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0167] In some possible implementations, the data generated during the operation is the activation value generated by the AI ​​model at each layer during at least one inference process.

[0168] In some possible implementations, the data in the first group are activation values ​​generated by at least one layer of the AI ​​model during inference, or activation values ​​generated by different layers of the same set of input data during the inference process of the AI ​​model.

[0169] In some possible implementations, the parameter for anomaly suppression is a scaling factor, and the outlier suppression module 202 is specifically used for:

[0170] When the degree of anomaly is less than a first threshold, the scaling factor of the data in the first group is determined to be a first value;

[0171] When the degree of abnormality is greater than or equal to the first threshold, the scaling factor of the activation value in the first group is determined based on the activation value and weight in the first group.

[0172] In some possible implementations, the outlier suppression module 202 is specifically used for:

[0173] When the degree of abnormality is less than or equal to the second threshold, the scaling factor of the activation value in the first group is determined according to the ratio of the maximum activation value to the maximum weight in the first group.

[0174] When the degree of anomaly is greater than the second threshold, a transfer coefficient is searched that makes the quantization error of the data in the first group meet the conditions, and a scaling factor for the activation value in the first group is determined based on the maximum activation value, the maximum weight and the transfer coefficient in the first group.

[0175] In some possible implementations, the system 200 further includes:

[0176] The calibration module 206 is used to sort the multiple groups according to the degree of abnormality of the data in each group, and restore the data in the target group of the multiple groups to the data before suppression according to the sorting result of the multiple groups. The target group is a group whose degree of abnormality is greater than a third threshold, or a group whose degree of abnormality is sorted from high to low as the top n groups.

[0177] Similar to outlier suppression module 202 and quantization module 204, calibration module 206 can be implemented in software or hardware. When implemented in software, calibration module 206 can be an application running on a computer device, such as a VM service, BMS service, or container service. When implemented in hardware, calibration module 206 can include at least one computing device, such as a server. Alternatively, calibration module 206 can also be a device implemented using an ASIC or a PLD.

[0178] In some possible implementations, the quantization module 204 is specifically used for:

[0179] When the abnormality of the data in the first group is less than the fourth threshold, the first proportion of abnormal values ​​in the suppressed data in the first group are discarded, and quantization is performed based on the maximum and minimum values ​​of the remaining data.

[0180] When the abnormality of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the data in the suppressed first group.

[0181] In some possible implementations, the quantization module 204 is specifically used for:

[0182] Obtain the random probability of the first group;

[0183] When the random probability is greater than the second value, the data in the suppressed first group is quantized.

[0184] In some possible implementations, the data in the first group includes activation values ​​and weights in the first group, and the quantization module 204 is specifically used for:

[0185] The weights in the first group after suppression are subjected to an exponential transformation to obtain the transformed weights.

[0186] The transformed weights are quantized, and the inverse exponential transformation of the quantized weights is performed.

[0187] In some possible implementations, the quantization module 204 is specifically used for:

[0188] The search term is the exponent that makes the quantization error of the weights in the first group after suppression satisfy the condition.

[0189] Based on the index, the weights in the suppressed first group are subjected to an exponential transformation.

[0190] This application also provides a computing device 800. For example... Figure 8 As shown, the computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other via the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0191] The 802 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus 802 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the computing device 800 (e.g., memory 806, processor 804, communication interface 808).

[0192] Processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0193] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). Memory 806 stores executable program code, which processor 804 executes to implement the aforementioned data quantization method. Specifically, memory 806 stores instructions for the data quantization system 200 to execute the data quantization method.

[0194] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0195] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0196] like Figure 9 As shown, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store instructions from the same data quantization system 200 for executing data quantization methods.

[0197] In some possible implementations, one or more computing devices 800 in the computing device cluster can also be used to execute some of the instructions used by the data quantization system 200 to execute the data quantization method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions used by the data quantization system 200 to execute the data quantization method.

[0198] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster can store different instructions for executing some functions of the data quantization system 200.

[0199] Figure 10 One possible implementation is shown. For example... Figure 10As shown, two computing devices 800A and 800B are connected via a communication interface 808. The memory in computing device 800A stores instructions for executing the functions of the outlier suppression module 202. The memory in computing device 800B stores instructions for executing the functions of the quantization module 204. Furthermore, the memory in computing device 800A also stores instructions for executing the functions of the calibration module 206. In other words, the memory 806 of computing devices 800A and 800B jointly stores the instructions used by the data quantization system 200 to execute the data quantization method.

[0200] Figure 10 The connection method between the computing device clusters shown is based on the consideration that the data quantization method provided in this application requires a large amount of computing power for quantization. Therefore, it is considered that the functions implemented by the quantization module 204 are performed by the computing device 800B.

[0201] It should be understood that Figure 10 The functions of the computing device 800A shown can also be performed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be performed by multiple computing devices 800.

[0202] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 One possible implementation is shown. For example... Figure 11 As shown, the two computing devices 800C and 800D are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 806 in computing device 800C stores instructions for performing the functions of the outlier suppression module 202. Simultaneously, the memory 806 in computing device 800D stores instructions for performing the functions of the quantization module 204. Furthermore, the memory in computing device 800C also stores instructions for performing the functions of the calibration module 206.

[0203] Figure 11 The connection method between the computing device clusters shown can be that, considering that the data quantization method provided in this application requires a lot of computing power for quantization, the function implemented by the quantization module 204 is handed over to the computing device 800D for execution.

[0204] It should be understood that Figure 11 The functions of the computing device 800C shown can also be performed by multiple computing devices 800. Similarly, the functions of the computing device 800D can also be performed by multiple computing devices 800.

[0205] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data quantization method described above in the data quantization system 200.

[0206] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the aforementioned data quantization method.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data quantification method, characterized in that, The method includes: Run an artificial intelligence (AI) model and acquire the data generated during the operation of the AI ​​model, the data including multiple groups; the multiple groups include a first group; Determine the degree of anomaly in the data of the first group, wherein the degree of anomaly indicates the degree of difference in the size of the data in the first group; Based on the degree of anomaly, parameters for anomaly suppression of the data in the first group are determined; The data in the first group are suppressed according to the parameters of the anomaly suppression to obtain the suppressed first group; The data in the first group after suppression are quantized.

2. The method according to claim 1, characterized in that, The data generated during the operation is the activation value generated by the AI ​​model at each layer during at least one inference process.

3. The method according to claim 1, characterized in that, The data in the first group are activation values ​​generated by at least one layer of the AI ​​model during the inference process, or activation values ​​generated by different layers of the same set of input data during the inference process of the AI ​​model.

4. The method according to any one of claims 1 to 3, characterized in that, The parameter for anomaly suppression is a scaling factor, and the parameter for determining anomaly suppression of the data in the first group based on the degree of anomaly includes: When the degree of abnormality is less than the first threshold, the scaling factor of the data in the first group is determined to be the first value; When the degree of abnormality is greater than or equal to the first threshold, the scaling factor of the activation value in the first group is determined based on the activation value and weight in the first group.

5. The method according to claim 4, characterized in that, The step of determining the scaling factor for the activation values ​​in the first group based on the activation values ​​and weights in the first group includes: When the degree of abnormality is less than or equal to the second threshold, the scaling factor of the activation value in the first group is determined according to the ratio of the maximum activation value to the maximum weight in the first group. When the degree of anomaly is greater than the second threshold, a transfer coefficient is searched that makes the quantization error of the data in the first group meet the conditions; a scaling factor for the activation value in the first group is determined based on the maximum activation value, the maximum weight, and the transfer coefficient in the first group.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The multiple groups are sorted according to the degree of anomaly in the data of each group; Based on the sorting results of the multiple groups, the data in the target group among the multiple groups is restored to the data before suppression. The target group is a group whose abnormality level is greater than the third threshold, or a group whose abnormality level is sorted from high to low as the top n groups.

7. The method according to any one of claims 1 to 6, characterized in that, The quantization of the data in the suppressed first group includes: When the abnormality of the data in the first group is less than the fourth threshold, the first proportion of abnormal values ​​in the suppressed data in the first group are discarded, and quantization is performed based on the maximum and minimum values ​​of the remaining data. When the abnormality of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the data in the suppressed first group.

8. The method according to any one of claims 1 to 7, characterized in that, The quantization of the data in the suppressed first group includes: Obtain the random probability of the first group; When the random probability is greater than the second value, the data in the suppressed first group is quantized.

9. The method according to any one of claims 1 to 8, characterized in that, The data in the first group includes activation values ​​and weights in the first group. The quantization of the suppressed data in the first group includes: The weights in the first group after suppression are subjected to an exponential transformation to obtain the transformed weights. The transformed weights are quantized, and the inverse exponential transformation of the quantized weights is performed.

10. The method according to claim 9, characterized in that, The step of performing an exponential transformation on the weights in the suppressed first group to obtain the transformed weights includes: The search term is the exponent that makes the quantization error of the weights in the first group after suppression satisfy the condition. Based on the index, the weights in the suppressed first group are subjected to an exponential transformation.

11. A data quantization system, characterized in that, The system includes: An outlier suppression module is used to run an artificial intelligence (AI) model, acquire data generated by the AI ​​model during operation, the data includes multiple groups, the multiple groups include a first group, determine the degree of anomaly of the data in the first group, the degree of anomaly indicates the degree of difference in the size of the data in the first group, determine parameters for anomaly suppression of the data in the first group based on the degree of anomaly, and suppress the data in the first group according to the anomaly suppression parameters to obtain the suppressed first group; The quantization module is used to quantize the data in the suppressed first group.

12. The system according to claim 11, characterized in that, The data generated during the operation is the activation value generated by the AI ​​model at each layer during at least one inference process.

13. The system according to claim 11, characterized in that, The data in the first group are activation values ​​generated by at least one layer of the AI ​​model during the inference process, or activation values ​​generated by different layers of the same set of input data during the inference process of the AI ​​model.

14. The system according to any one of claims 11 to 13, characterized in that, The parameter for anomaly suppression is a scaling factor, and the outlier suppression module is specifically used for: When the degree of abnormality is less than the first threshold, the scaling factor of the data in the first group is determined to be the first value; When the degree of abnormality is greater than or equal to the first threshold, the scaling factor of the activation value in the first group is determined based on the activation value and weight in the first group.

15. The system according to claim 14, characterized in that, The outlier suppression module is specifically used for: When the degree of abnormality is less than or equal to the second threshold, the scaling factor of the activation value in the first group is determined according to the ratio of the maximum activation value to the maximum weight in the first group. When the degree of anomaly is greater than the second threshold, a transfer coefficient is searched that makes the quantization error of the data in the first group meet the conditions, and a scaling factor for the activation value in the first group is determined based on the maximum activation value, the maximum weight and the transfer coefficient in the first group.

16. The system according to any one of claims 11 to 15, characterized in that, The system also includes: The calibration module is used to sort the multiple groups according to the degree of anomaly of the data in each group, and restore the data in the target group of the multiple groups to the data before suppression according to the sorting result of the multiple groups. The target group is a group whose anomaly degree is greater than a third threshold, or a group whose anomaly degree is sorted into the top n from high to low.

17. The system according to any one of claims 11 to 16, characterized in that, The quantization module is specifically used for: When the abnormality of the data in the first group is less than the fourth threshold, the first proportion of abnormal values ​​in the suppressed data in the first group are discarded, and quantization is performed based on the maximum and minimum values ​​of the remaining data. When the abnormality of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the data in the suppressed first group.

18. The system according to any one of claims 11 to 17, characterized in that, The quantization module is specifically used for: Obtain the random probability of the first group; When the random probability is greater than the second value, the data in the suppressed first group is quantized.

19. The system according to any one of claims 11 to 18, characterized in that, The data in the first group includes the activation values ​​and weights in the first group, and the quantization module is specifically used for: The weights in the first group after suppression are subjected to an exponential transformation to obtain the transformed weights. The transformed weights are quantized, and the inverse exponential transformation of the quantized weights is performed.

20. The system according to claim 19, characterized in that, The quantization module is specifically used for: The search term is the exponent that makes the quantization error of the weights in the first group after suppression satisfy the condition. Based on the index, the weights in the suppressed first group are subjected to an exponential transformation.

21. A computer program product, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the data quantization method according to any one of claims 1 to 10.

Images