Data quantization method and related device

Abstract

Provided is a data quantization method, comprising: running an artificial intelligence (AI) model, and obtaining data generated in the operation process of the AI model, wherein the data comprises a plurality of groups; determining an abnormality degree of data in a first group among the plurality of groups, wherein the abnormality degree indicates the difference degree of the size of the data in the first group; on the basis of the abnormality degree, determining parameters for performing abnormality suppression on the data in the first group; on the basis of the parameters for the abnormality suppression, suppressing the data in the first group; and quantizing the suppressed data in the first group. By respectively determining the abnormality degrees of different groups, and selecting for different groups appropriate parameters for abnormality suppression on the basis of the abnormality degrees so as to suppress abnormality values, adaptive suppression of data in different groups is achieved, thereby avoiding the situation of insufficient suppression or excessive suppression, and improving the quantization precision of the AI model.

Description

A data quantification method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 29, 2024, with application number 202410231737.8 and invention name “A data quantization method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a data quantization method, a data quantization system, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the continuous development of artificial intelligence (AI) technology, the parameter size of AI models (also referred to as models) has continued to increase. For example, the parameter size of large-scale language models (LLMs), which are currently attracting widespread attention, can reach billions, tens of billions, hundreds of billions, or even trillions of parameters. Compared with conventional models, large models (such as the aforementioned LLMs) have stronger generalization capabilities and better performance on various tasks, thus having broad application prospects.

[0004] While increasing parameter size can improve the performance and generalization capabilities of AI models, it also poses challenges to their deployment. Currently, the industry primarily uses model quantization to compress AI models. Deploying compressed AI models reduces memory usage and computational complexity during inference, thereby accelerating inference and reducing inference costs.

[0005] However, large models have outliers in their activation distribution compared to conventional models. Activation distribution refers to the distribution of model activation values. Activation values ​​can be features of the AI ​​model's inputs and intermediate layers. Outliers are outliers in the activation distribution, typically extreme values ​​in the activation distribution. Outliers can render conventional quantization algorithms ineffective. Forcing outliers to zero can render the AI ​​model's accuracy completely unusable. Therefore, outliers are crucial to the AI ​​model's capabilities. Providing a high-precision, highly available model quantization solution has become a key concern in the industry.

[0006] Summary of the Invention

[0007] This application provides a data quantization method that determines the degree of abnormality of the data in different groups generated during the operation of an AI model. Based on the degree of abnormality, it determines parameters for suppressing abnormalities in the data in the corresponding groups. Based on the parameters, it adaptively suppresses abnormalities in different groups and quantizes the suppressed data. This method can achieve high-precision and high-availability model quantization. This application also provides a data quantization system, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above-mentioned data quantization method.

[0008] In a first aspect, the present application provides a data quantization method. The method can be executed by a data quantization system. The data quantization system can be a software system. The software system can be integrated with a model compression tool or a model deployment tool, for example, as a functional module of a model compression tool or a model deployment tool, or integrated with a model compression tool or a model deployment tool in the form of a plug-in. The software system can also be an independent software system. The software system can be provided to users in the form of a software package, and deployed by the user in a local data center or a private cloud. The software system can also be deployed in a public cloud and provided to users in the form of a cloud service. The data quantization system can also be a hardware system, for example, a computing device cluster with data quantization capabilities, which executes the data quantization method of the present application when the computing device cluster is running.

[0009] Specifically, the data quantization system can run an AI model to obtain data generated during the AI ​​model's operation, the data comprising multiple groups, and then determine the degree of abnormality of the data in a first group of the multiple groups, where the degree of abnormality indicates the degree of difference in the size of the data in the first group. The data quantization system can then determine parameters for suppressing abnormalities in the data in the first group based on the degree of abnormality, suppress the data in the first group based on the abnormality suppression parameters, and quantify the suppressed data in the first group.

[0010] This method determines the degree of abnormality for different groups separately, and selects appropriate abnormality suppression parameters for different groups based on the degree of abnormality to suppress outliers, thereby achieving adaptive suppression of data in different groups, thereby avoiding insufficient or excessive suppression and improving the quantitative accuracy of the AI ​​model.

[0011] In some possible implementations, the data generated during operation are activation values ​​generated by each layer of the AI ​​model during at least one inference process. Considering the randomness of the distribution of activation values ​​generated by each layer during a single inference process, the present application can sample the activation values ​​generated by each layer of the AI ​​model during multiple inference processes. Based on statistical analysis of the activation values ​​generated by each layer during multiple inference processes, more representative activation values ​​can be obtained. The parameters thus determined for suppressing abnormal activation values ​​are more accurate, resulting in a better suppression effect.

[0012] In some possible implementations, the data in the first group may be activation values ​​generated by at least one layer in the AI ​​model during inference, or may be activation values ​​generated by different layers of the same set of input data during inference. The same set of input data may be input data of at least one channel. For example, when the input data is an image, the data in the first group may be activation values ​​generated by different layers of the input data of the R channel during inference on the AI ​​model.

[0013] This method supports grouping based on layer granularity or channel granularity to achieve adaptive suppression of data in different layers or data in different channels.

[0014] In some possible implementations, the parameter for anomaly suppression is a scaling factor. Accordingly, when the degree of anomaly is less than a first threshold, the data quantization system may determine the scaling factor of the data in the first group to be a first value; when the degree of anomaly is greater than or equal to the first threshold, the data quantization system may determine the scaling factor of the activation values ​​in the first group based on the activation values ​​and weights in the first group.

[0015] This method can determine the scaling factor in different ways according to the degree of abnormality, thereby achieving adaptive suppression of different groups according to the degree of abnormality, avoiding insufficient or excessive suppression affecting the quantitative accuracy of the AI ​​model.

[0016] In some possible implementations, when the degree of abnormality is less than or equal to a second threshold, the data quantization system determines a scaling factor for the activation values ​​in the first group based on the ratio of the maximum activation value to the maximum weight in the first group. When the degree of abnormality is greater than the second threshold, a transfer coefficient is searched for that satisfies a quantization error of the data in the first group, and the scaling factor for the activation values ​​in the first group is determined based on the maximum activation value, the maximum weight, and the transfer coefficient in the first group.

[0017] In this method, the situation where the degree of abnormality is greater than or equal to the first threshold can be further divided into: greater than or equal to the first threshold and less than or equal to the second threshold; greater than the second threshold. When the degree of abnormality of the data in the first group is greater than or equal to the first threshold and less than or equal to the second threshold, it means that the abnormality range is in the middle. The data quantization system can directly balance the distribution of activation values ​​and weights to determine the scaling factor. When the degree of abnormality of the data in the first group is greater than the second threshold, it means that the abnormality is large. The data quantization system can introduce a transfer coefficient, also known as the α coefficient, to determine the scaling factor. In this way, fine-grained abnormality suppression can be achieved to meet business needs.

[0018] In some possible implementations, the data quantification system may further sort the multiple groups based on the degree of abnormality of the data in each of the multiple groups, and based on the sorting results, restore the data in a target group among the multiple groups to the data before suppression. The target group is a group whose abnormality level is greater than a third threshold, or a group ranked in the top n in descending order of abnormality level.

[0019] This method takes into account that the data range of some network layers is very large. In order to maximize the quantitative accuracy of the AI ​​model, groups with larger data ranges can be identified and their data, such as activation values, can be rolled back to achieve lossless accuracy as much as possible.

[0020] In some possible implementations, when the degree of abnormality of the data in the first group is less than a fourth threshold, the data quantization system may discard a first proportion of abnormal values ​​in the suppressed data in the first group and perform quantization based on the maximum and minimum values ​​of the remaining data; when the degree of abnormality of the data in the first group is greater than or equal to the fourth threshold, the data quantization system may perform quantization based on the maximum and minimum values ​​of the data in the suppressed first group.

[0021] The method can adaptively select a quantization algorithm that matches the abnormality level of the data in the first group according to the abnormality level of the data, thereby achieving a balance between the ability to resist outlier interference and accuracy.

[0022] In some possible implementations, the data quantization system can also obtain a random probability for the first group. When the random probability is greater than a second value, the data quantization system can quantize the suppressed data in the first group. Compared to no quantization, random quantization is more consistent with real-world scenarios, and compared to full quantization, random quantization also avoids overfitting, resulting in better generalization accuracy.

[0023] In some possible implementations, the data in the first group includes activation values ​​and weights in the first group. The data quantization system may perform an exponential transformation on the suppressed weights in the first group to obtain transformed weights, then quantize the transformed weights, and perform an inverse transformation of the exponential transformation on the quantized weights.

[0024] This method introduces an exponential function to equivalently transform the quantization formula to obtain a non-uniform, adaptive quantization resolution, which can more effectively quantize weights containing outliers. The non-uniform quantization in this application is an element-by-element operation. Compared to other non-uniform quantization methods based on table lookup or shifting, this non-uniform quantization can be implemented without special hardware and at a low cost.

[0025] In some possible implementations, the data quantization system may search for an exponent that satisfies a condition for the quantization error of the weights in the suppressed first group, and then perform an exponential transformation on the weights in the suppressed first group based on the exponential transformation. This can minimize the error introduced by the exponential transformation.

[0026] In a second aspect, the present application provides a data quantization system. The system comprises:

[0027] an outlier suppression module, configured to run an artificial intelligence (AI) model, obtain data generated during the running of the AI ​​model, the data including a plurality of groups, determine a degree of abnormality of data in a first group of the plurality of groups, the degree of abnormality indicating a degree of difference in size of the data in the first group, determine parameters for performing abnormality suppression on the data in the first group based on the degree of abnormality, and suppress the data in the first group based on the abnormality suppression parameters;

[0028] The quantization module is used to quantize the data in the suppressed first group.

[0029] In some possible implementations, the data generated during the operation is the activation value generated by each layer of the AI ​​model during at least one inference process.

[0030] In some possible implementations, the data in the first group are activation values ​​generated by at least one layer in the AI ​​model during the inference process, or are activation values ​​generated by different layers in the inference process of the AI ​​model for the same set of input data.

[0031] In some possible implementations, the anomaly suppression parameter is a scaling factor, and the outlier suppression module is specifically configured to:

[0032] When the abnormality level is less than a first threshold, determining a scaling factor of the data in the first group to be a first value;

[0033] When the abnormality level is greater than or equal to the first threshold, a scaling factor of the activation values ​​in the first group is determined according to the activation values ​​and weights in the first group.

[0034] In some possible implementations, the outlier suppression module is specifically configured to:

[0035] When the abnormality level is less than or equal to a second threshold, determining a scaling factor of the activation value in the first group according to a ratio of a maximum activation value to a maximum weight in the first group;

[0036] When the degree of abnormality is greater than the second threshold, a transfer coefficient is searched so that the quantization error of the data in the first group meets the condition, and a scaling factor of the activation value in the first group is determined according to the maximum activation value, the maximum weight and the transfer coefficient in the first group.

[0037] In some possible implementations, the system further includes:

[0038] A calibration module is configured to sort the multiple groups according to the degree of abnormality of the data in each of the multiple groups, and restore the data in a target group among the multiple groups to the data before suppression based on the sorting results of the multiple groups, where the target group is a group whose abnormality is greater than a third threshold, or a group ranked in the top n in descending order of abnormality.

[0039] In some possible implementations, the quantization module is specifically configured to:

[0040] When the abnormality degree of the data in the first group is less than a fourth threshold, discarding a first proportion of abnormal values ​​from the suppressed data in the first group, and quantizing the remaining data according to the maximum and minimum values;

[0041] When the abnormality degree of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the suppressed data in the first group.

[0042] In some possible implementations, the quantization module is specifically configured to:

[0043] Obtaining a random probability of the first group;

[0044] When the random probability is greater than a second value, quantizing the data in the suppressed first group.

[0045] In some possible implementations, the data in the first group includes activation values ​​and weights in the first group, and the quantization module is specifically configured to:

[0046] Performing exponential transformation on the weights in the suppressed first group to obtain transformed weights;

[0047] The transformed weights are quantized, and an inverse transform of the exponential transform is performed on the quantized weights.

[0048] In some possible implementations, the quantization module is specifically configured to:

[0049] Searching for an index that satisfies a condition for the quantization error of the weights in the first group after suppression;

[0050] The weights in the suppressed first group are exponentially transformed according to the exponent.

[0051] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, each of which includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, causing the computing device or computing device cluster to perform the data quantization method described in the first aspect or any implementation of the first aspect.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the data quantization method described in the first aspect or any implementation of the first aspect.

[0053] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the data quantization method described in the first aspect or any one of the implementations of the first aspect.

[0054] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0056] FIG1 is a schematic diagram of the activation value distribution of different network layers of an AI model provided by this application;

[0057] FIG2 is a schematic diagram of the architecture of a data quantization system provided by this application;

[0058] FIG3 is a schematic diagram of a process for suppressing outliers provided by the present application;

[0059] FIG4 is a schematic diagram of a process of post-training quantization calibration provided by the present application;

[0060] FIG5 is a schematic diagram showing a comparison between uniform quantization and non-uniform quantization provided by the present application;

[0061] FIG6 is a flow chart of a data quantification method provided by the present application;

[0062] FIG7 is a schematic diagram of a quantization interface provided by the present application;

[0063] FIG8 is a schematic diagram of the structure of a computing device provided by the present application;

[0064] FIG9 is a schematic diagram of the structure of a computing device cluster provided by the present application;

[0065] FIG10 is a schematic diagram of the structure of another computing device cluster provided by the present application;

[0066] FIG11 is a schematic diagram of the structure of another computing device cluster provided in this application. DETAILED DESCRIPTION

[0067] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0068] First, some technical terms involved in the embodiments of this application are introduced.

[0069] An AI model is a model built based on AI technology. AI technology may include, but is not limited to, machine learning (ML) and deep learning (DL). Accordingly, an AI model may be a neural network model trained through machine learning or deep learning.

[0070] An AI model can include multiple network layers, such as convolutional layers, pooling layers, and fully connected layers. The parameters in a network layer used to perform calculations with the input of that network layer include weights, also known as weight values. The input of the AI ​​model and the features of the intermediate layers constitute the activation value of the AI ​​model. The features of the intermediate layers can be the features extracted by the network layers between the input layer and the output layer, such as the convolutional layer and the pooling layer. The activation value or weight can include at least one channel, where the channel of the activation value or weight can be understood as the dimension of the activation value or weight. For example, if the AI ​​model is a neural network model for image processing, the input of the AI ​​model can be an image, which can include three dimensions: red, green, and blue. Based on this, the activation value can include three channels. Similarly, the weights of the convolutional layer in the AI ​​model can include the elements of the convolution kernel. The weights can include input channels and output channels, where the input channels are typically the same as the output channels of the previous layer, and the number of output channels can be equal to the number of convolution kernels.

[0071] Model quantization refers to the introduction of quantization parameters, such as scaling factors and biases (also known as offsets), to convert floating-point values ​​in AI models into fixed-point values ​​to achieve AI model compression and inference acceleration. Floating-point values ​​in AI models can include activation values ​​and weights, and model quantization can include quantization of activation values ​​and quantization of weights.

[0072] Post-training Quantization (PTQ) converts a pre-trained floating-point model directly into a fixed-point model. The floating-point model can be in single-precision (Full Precise Float 32, FP32) or half-precision (float, FP16) format, while the fixed-point model can be an integer-based model. Fixed-point models retain integers and decimals to a specific number of digits, while floating-point models retain significands and exponents to a specific number of digits. This process eliminates the need to train the original model (e.g., the pre-trained floating-point model). The quantized parameters can be calibrated or optimized to ensure that the accuracy loss after quantization meets the requirements.

[0073] Model quantization can include two parts: quantization for weights and quantization for activations. In some examples, 8-bit quantization can be performed on weights and activations, which is also called W8A8, thereby achieving equivalent 32-bit performance. W represents weights and A represents activations. Specifically, the basic operations in neural networks include convolution operations of weights and activations, denoted as W*A, where convolution operations can include multiplication and addition operations. If one of the items is quantized to {-1,1}, the multiplication and addition operations can be simplified to addition and subtraction operations. If both items are quantized to {-1,1}, the multiplication and addition operations are simplified to bitwise operations, which are very friendly to hardware calculations.

[0074] In order to implement the deployment of AI models, especially on resource-constrained end-side devices, model quantization such as PTQ is usually used to compress the AI ​​models. However, as the parameter scale of the AI ​​model increases, the activation values ​​of the AI ​​model begin to show outliers, which can make conventional quantization algorithms ineffective. Relevant data shows that the activation values ​​of large models with a parameter scale greater than 6.7 billion (denoted as 6.7 billion, abbreviated as 6.7B) begin to show outliers. Although the proportion of outliers is only 1%, the amplitude range is widely distributed, for example, the amplitude range can range from hundreds to thousands.

[0075] For ease of understanding, this application also provides an example for illustration. Figure 1 shows a schematic diagram of the activation amplitude distribution of each network layer of a large model. The parameter scale of this large model is 70B, and the number of layers in this large model is over 500. The activation amplitude of some network layers far exceeds that of other network layers, and the outlier phenomenon is relatively significant, which is not conducive to quantification.

[0076] Forcing the 1% of outliers mentioned above to zero can completely destroy the accuracy of the AI ​​model, so outliers are crucial to the performance of the AI ​​model. To address this, the industry has proposed solutions to suppress outliers. Currently, the mainstream outlier suppression solution is to reduce outliers in activation values ​​through equivalent transformations. This solution introduces an offset factor (denoted as shift) and a scaling factor (denoted as scale) to reduce outliers.

[0077] The specific steps are as follows: 1. An offset factor shift is introduced into the activation value of each network layer so that the activation values ​​on all network layers are migrated to a range centered on 0; 2. A scaling interval value t that minimizes the quantization error is calculated through grid search. Based on this scaling interval value t, the corresponding scaling factor scale can be derived; 3. The activation values ​​on all network layers are offset and scaled.

[0078] However, the scheme of determining scale based on grid search tends to narrow the range of all activation values. In order to achieve an equivalent transformation, the weight values ​​usually need to be inversely transformed. For example, to multiply the activation value by the scaling factor scale, the weight needs to be divided by the scaling factor scale. This makes the weight distribution more rugged, which is not conducive to the quantization of smaller models, nor is it conducive to the quantization of some network layers with smooth activation value distribution in large models. In addition, the amplitude range of outliers in different network layers is widely distributed. If a single suppression method is used, there may be insufficient suppression (the outliers after suppression are still significant, making the quantization algorithm ineffective) or excessive suppression (the activation values ​​after suppression are easy to quantize, but the difficulty of quantizing the weights increases significantly. This phenomenon is also called over-transfer). Both insufficient and excessive suppression can lead to reduced accuracy of the AI ​​model.

[0079] In view of this, the present application provides a data quantization method. This method takes into account the fact that the size distribution range of the data generated by the AI ​​model during operation is relatively wide. For example, the amplitude range of the outliers in the activation values ​​of different network layers or different channels of the AI ​​model during inference is relatively wide. Using a single suppression method may result in insufficient or excessive suppression. A strategy for grouping adaptive suppression of outliers is proposed. Specifically, the data generated during the operation of the AI ​​model may include multiple groups. For example, the activation value or weight of each network layer can form a group. Based on the degree of abnormality of the data in the group, the parameters for suppressing abnormalities in the data in the group are determined, and then the data in the first group is suppressed according to the parameters for abnormal suppression, and the data in the suppressed first group is quantized. This method achieves adaptive suppression of data in different groups by determining the degree of abnormality for different groups respectively, and selecting appropriate abnormality suppression parameters for different groups based on the degree of abnormality to suppress outliers. This can avoid insufficient or excessive suppression and improve the quantization accuracy of the AI ​​model.

[0080] The data quantization method of the present application can be applied to scenarios of AI model compression or AI model deployment, especially scenarios of large model compression or deployment. Specifically, large models can be deployed on resource-constrained devices such as the end side, or on the cloud side for tasks such as dialogue and language understanding. The data quantization method of the present application is used to quantify the data during the operation of the above-mentioned large model, such as the activation values ​​and other data generated during the large model inference process and the weights of the large model. This can reduce deployment costs, reduce the amount of computation of the AI ​​model, achieve inference acceleration, and effectively improve inference performance.

[0081] The method can be performed by a data quantization system. The data quantization system can be a software system. The software system can be integrated with a model compression tool or a model deployment tool, for example, as a functional module of a model compression tool or a model deployment tool, or integrated with a model compression tool or a model deployment tool in the form of a plug-in. The software system can also be an independent software system. Among them, the software system can be provided to users in the form of a software package, and deployed by the user on a stand-alone computing device. The software system can also be deployed in a public cloud and provided to users in the form of a cloud service. Among them, when the software system is deployed in a public cloud, it can be deployed in a virtual machine in the public cloud, or distributedly deployed in different virtual machines in the public cloud. The data quantization system can also be a hardware system, for example, a computing device cluster with data quantization capabilities, which executes the data quantization method of the present application when the computing device cluster is running.

[0082] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the data quantization system of the present application is first introduced with reference to the accompanying drawings.

[0083] Referring to FIG2 , which illustrates an architecture diagram of a data quantization system, data quantization system 200 is used to quantize data during the operation of an AI model, for example, to quantize activation values ​​and / or AI model weights generated during AI model inference. The AI ​​model can be a fully trained model, such as a pre-trained model, which can be a large model. In some examples, the AI ​​model can also be a small model.

[0084] The data quantization system 200 may include an outlier suppression module 202 and a quantization module 204. Furthermore, the data quantization system 200 may also include a calibration module 206. The calibration module 206 may implement adaptive PTQ calibration, and the quantization module 204 may implement quantization of data such as activation values ​​or weights. Before quantization or calibration, the data quantization system 200 may first use the outlier suppression module 202 to perform an equivalent transformation on the AI ​​model to suppress outliers in the data, and then input the suppressed data into the calibration module 206 for activation value fallback to achieve adaptive calibration. The calibrated data may then be input into the quantization module 204 for quantization. For activation values, the quantization module 204 may use random quantization, and for weights, the quantization module 204 may use low-bit non-uniform quantization.

[0085] The functions of the outlier suppression module 202 , the quantization module 204 , and the calibration module 206 are described in detail below.

[0086] The outlier suppression module 202 is used to run the AI ​​model, obtain data generated during the operation of the AI ​​model, and determine the degree of abnormality of the data in the first group among the multiple groups. The degree of abnormality indicates the difference in length of the size of the data in the first group. Then, according to the degree of abnormality, parameters for suppressing abnormality of the data in the first group are determined, and the data in the first group is suppressed according to the parameters for suppressing abnormality.

[0087] Among them, the data generated during the operation process can be the activation value generated by each layer of the AI ​​model during at least one reasoning process. The data in the first group can be the data generated by at least one layer in the AI ​​model during the reasoning process, such as the activation value generated by at least one layer in the AI ​​model during the reasoning process. In some examples, the data in the first group can be the activation values ​​generated by different layers of the same set of input data during the AI ​​model reasoning process. The input data may include multiple channels. Taking an image as an example, the input data may include three channels of R, G, and B. Data of different channels can generate activation values ​​at different layers during the AI ​​model reasoning process. The data in the first group can be the activation values ​​generated by data of at least one channel at different layers during the AI ​​model reasoning process. For ease of description, the data in the first group is taken as the activation value generated by at least one layer in the AI ​​model during the reasoning process.

[0088] Since the degree of abnormality of data in different groups can be different, if a single suppression method is used, there may be insufficient or excessive suppression, which may lead to reduced accuracy of the AI ​​model. To this end, the outlier suppression module 202 of the present application can use different methods to determine the parameters of abnormality suppression for groups with different abnormality levels to improve the outlier suppression effect.

[0089] The outlier suppression parameters may include a scale factor (scale) and, further, a shift factor (shift). Taking activation value outlier suppression as an example, outlier suppression can be divided into two steps: the first step is to introduce the shift factor (Shift) to offset the activation value, and the second step is to introduce the scale factor (scale) to scale the activation value.

[0090] The activation value offset factor Shift can be the average of the maximum activation value and the minimum activation value in the first group. The activation value group X is formed by taking the first group as the activation value of the i-th layer of the AI ​​model. i For example, the activation value offset factor can be determined by the following formula:

[0091] Among them, Shift i Characterizes the activation value offset factor.

[0092] By performing the shift according to the activation value shift factor, the activation values ​​in the activation value group can be migrated to a range symmetrically centered around 0, thereby minimizing the value range of the activation value group.

[0093] Regarding the scaling factor Scale, this application takes into account that the degree of abnormality of outliers in different groups (for example, outliers in different network layers) may be different, and introduces a factor to measure the degree of abnormality, also called the degree of abnormality, denoted as Range. The outlier suppression module 202 can determine the scaling factor of the data in the first group based on the degree of abnormality of the data in the first group.

[0094] The abnormality level indicates the degree of difference in the size of the data in the first group. In some examples, the abnormality level can be the ratio of the maximum absolute value of the activation value in the activation value group to the standard deviation. Specifically, it can be determined according to the following formula:

[0095] Among them, Range i Represents the activation value group X i Abnormal degree, abs represents the absolute value, std represents the standard deviation. Range i The larger the value, the more activated the value group X is. iThe higher the abnormality.

[0096] In a specific implementation, the outlier suppression module 202 can model the scaling factor and the degree of outlier to obtain a mapping relationship between the scaling factor and the degree of outlier. This mapping relationship can be represented by a function. In some examples, the function can be a piecewise function. When the Range range is different, the scaling factor Scale can be calculated in different ways to adapt to the activation value quantization under different activation distributions.

[0097] Specifically, the outlier suppression module 202 is used to suppress the data in the first group (such as the activation value group X i When the abnormality level of the activation values ​​in the first group is less than a first threshold, the scaling factor of the data in the first group is determined to be the first value. When the abnormality level of the first group is greater than or equal to the first threshold, the scaling factor of the activation values ​​in the first group is determined based on the activation values ​​and weights in the first group. The first value can be a fixed value, such as 1, or another fixed value close to 1. The first threshold can be set based on experience and is not limited in this embodiment.

[0098] Furthermore, the case where the degree of abnormality is greater than or equal to the first threshold can be further divided into the following multiple cases: ① greater than or equal to the first threshold and less than or equal to the second threshold; ② greater than the second threshold. The abnormal value suppression module 202 is used to determine the scaling factor of the activation value in the first group according to the ratio of the maximum activation value to the maximum weight in the first group when the degree of abnormality of the data in the first group is greater than or equal to the first threshold and less than or equal to the second threshold; when the degree of abnormality of the data in the first group is greater than the second threshold, search for a transfer coefficient that satisfies the quantization error of the data in the first group, and determine the scaling factor of the activation value in the first group according to the maximum activation value, maximum weight and transfer coefficient in the first group.

[0099] The following is an explanation with reference to specific examples. As shown in Figure 3, the outlier suppression module 202 divides the Range into three levels according to the Range range, and provides a corresponding Scale calculation method for each level. In the first level, the Range range is the smallest. In this case, it can be considered that there are no outliers or the influence of the outliers is small (can be ignored), so there is no need to suppress outliers. In this case, the Scale can be set to 1; in the second level, the Range range is centered. In this case, the distribution of activation values ​​and weights can be directly balanced to obtain the Scale; in the third level: the Range range is the largest and the degree of abnormality is the largest. A transfer coefficient, also called the α coefficient, can be introduced. By setting the range of the α coefficient, for example, a range of 0-1 (including endpoint values), and then using a fine-grained search method, the optimal α coefficient can be solved to ensure that the quantization error is minimized. Then, the Scale is determined based on the solved α coefficient, the activation values ​​in the first group, and the weights in the first group. Among them, the quantization error can be the mean square error (MSE).

[0100] The scales corresponding to the above three ranges can be determined by the following formula:

[0101] Among them, Scale i The scaling factor representing the activation values ​​in the first group, for example, the activation value group X i The scaling factor, W i Represents the weight in the first group, for example, the weight group, and α represents the transfer coefficient. Range i Representing the activation value group X i The abnormality level, M1 is the first threshold, M2 is the second threshold.

[0102] The outlier suppression module 202 can scale the data in the first group (e.g., activation values) according to the scaling factor of the data in the first group to obtain the suppressed first group. After the outlier suppression module 202 uses the scaling factor to suppress the activation values ​​in the first group, it can also divide the weights in the first group by the scaling factor to achieve an equivalent transformation. In this way, outlier suppression can be performed on the data during the operation of the AI ​​model based on data distribution bins, avoiding the situation where insufficient or excessive suppression occurs due to the use of a single suppression method, thereby helping to improve quantization accuracy.

[0103] The quantization module 204 is used to quantize the data in the suppressed first group. The granularity of data quantization can be the same as the granularity of outlier suppression. For example, the granularity of activation value quantization can be group-by-group quantization, specifically quantization is performed in units of activation value groups, and an activation value group shares quantization parameters.

[0104] In some possible implementations, after suppressing outliers in the data generated by an AI model (such as a large model) during operation, the impact of outliers may not be completely eliminated. Based on this, in addition to the above-mentioned outlier suppression scheme, this application also provides an accuracy maintenance strategy during the calibration phase, including but not limited to an activation value fallback strategy.

[0105] Specifically, the first group includes activation values ​​in at least one network layer of the AI ​​model. Considering that the activation value range of some network layers is very large, in order to maximize the quantization accuracy of the AI ​​model, this application introduces activation value fallback. Activation value fallback refers to the inverse processing of the activation value after outlier suppression, restoring it to the activation value before outlier suppression.

[0106] Specifically, referring to the schematic diagram of a data calibration and quantization process shown in FIG4 , the calibration module 206 is used to sort the multiple groups according to the degree of abnormality of the data in each of the multiple groups, for example, sorting them in descending order to obtain the sorting results of the multiple groups. Then, based on the sorting results of the multiple groups, the data in the target group among the multiple groups is restored to the data before suppression, for example, the activation value in the target group is reverted to the activation value before suppression. The target group can be the top n groups ranked from high to low in terms of abnormality, or the group whose abnormality is greater than a third threshold. Related research has shown that the above-mentioned value of n is 1 to 5 to achieve lossless accuracy. The third threshold can be set based on empirical values. For example, the calibration module 206 can revert the activation values ​​of the top 5 network layers ranked by abnormality, so that the activation values ​​of the top 5 network layers ranked by abnormality are restored to the activation values ​​before suppression. In this way, the characteristics of large model quantization can be combined to calculate the layer-level quantization sensitivity with O(n) computational complexity, which is more efficient than traditional mixed precision algorithms.

[0107] Furthermore, when quantizing the data in the suppressed first group, the present application supports multiple quantization algorithms. The quantization module 204 can select a corresponding algorithm for quantization based on the degree of abnormality of the data in the first group. Specifically, when the degree of abnormality of the data in the first group is less than a fourth threshold, the quantization module 204 discards a first proportion of abnormal values ​​from the suppressed first group and quantizes the data based on the maximum and minimum values ​​of the remaining data; when the degree of abnormality of the data in the first group is greater than or equal to the fourth threshold, the quantization module 204 quantizes the data based on the maximum and minimum values ​​of the suppressed first group.

[0108] Let's use activation value quantization as an example. There are many activation value quantization algorithms, such as minimum-maximum quantization and histogram quantization. Minimum-maximum quantization counts the minimum and maximum values ​​of multiple sets of data and directly uses them as floating-point ranges. In the case of large outliers, the outliers are also treated as minimum or maximum values. Histogram quantization performs histogram modeling on multiple sets of data, discarding the first proportion of outliers and taking the middle distribution as the minimum and maximum values. It has strong resistance to outlier interference. When the outliers are small, the range calculated based on the modeling method is more accurate than the minimum-maximum method. When the outliers are large, the quantization method based on histogram modeling will directly eliminate the outliers, resulting in a serious decrease in accuracy.

[0109] To this end, the present application supports adaptive selection of activation value quantization algorithms based on data distribution. The quantization module 204 uses two algorithms, the minimum-maximum and histogram, to calculate the range of statistical activation values. When performing activation value quantization, the optimal algorithm is adaptively selected based on the data distribution. When the Range is less than the fourth threshold, the quantization module 204 can select histogram quantization to quantize the activation values ​​in the suppressed first group. When the Range is greater than or equal to the fourth threshold, the quantization module 204 can select minimum-maximum quantization to quantize the activation values ​​in the suppressed first group.

[0110] Since different activation value quantization algorithms are suitable for activation values ​​of different data distributions, adaptively selecting the optimal algorithm based on the characteristics of the activation distribution of the AI ​​model can improve quantization accuracy.

[0111] During the layer-by-layer quantization calibration process, random probability can be used to determine whether to quantize the data (e.g., activation values) of all preceding layers of each layer. In specific implementations, the quantization module 204 can obtain the random probability of the first group and, when the random probability is greater than a second value, quantize the suppressed data in the first group. Compared to no quantization, random quantization is more consistent with actual scenarios, and compared to full quantization, random quantization can also avoid overfitting. Therefore, random quantization has better generalization accuracy.

[0112] The activation value quantization process is described in detail above. The quantization process can also include weight quantization. The weight quantization process is described in detail below.

[0113] The quantization module 204 is also used to quantize the weights of the AI ​​model. The weights of the AI ​​model, such as the weights of the large model, are unevenly distributed and contain a certain proportion of outliers, among which the maximum weight can be more than 100 times the normal weight, which poses a challenge to low-bit quantization below 8 bits. As shown in Figure 5, ordinary uniform quantization lacks adaptive differentiation capability in its quantization resolution, which will result in a large amount of waste of precision bits, while non-uniform quantization can significantly improve accuracy. Based on this, the present application also provides an outlier adaptive method to achieve high-precision low-bit non-uniform quantization.

[0114] In this application, the core idea of ​​non-uniform quantization is to introduce an exponential function to equivalently transform the quantization formula to obtain a non-uniform and adaptive quantization resolution, so that the weights containing outliers can be quantized more friendly. Among them, this application can use the method of minimizing the error to solve the exponent of the exponential transformation of each weight. The non-uniform quantization of this application is an element-by-element operation. Compared with other non-uniform quantization based on table lookup or shifting, the non-uniform quantization of this application can be implemented without special hardware and at a low cost.

[0115] The non-uniform transformation of the present application is described in detail below.

[0116] In a specific implementation, the quantization module 204 is used to perform an exponential transformation on the weights in the suppressed first group to obtain the transformed weights, and then quantize the transformed weights, and perform an inverse transformation of the exponential transformation on the quantized weights to achieve equivalence before and after the transformation. The weights in the first group may include weights in at least one channel of the AI ​​model, or the first group may include weights in at least one network layer of the AI ​​model. For example, the weights of each network layer of the AI ​​model may form a weight group. For another example, the weights in each channel of each network layer of the AI ​​model may form a weight group. In some cases, the weights of multiple network layers or multiple channels with the same or similar distribution may also form a weight group.

[0117] Taking the exponential transformation of the weights in the first group as an example, the quantization module 204 is specifically configured to search for an exponent that satisfies a condition for the quantization error of the weights in the first group after being suppressed, and then perform exponential transformation on the weights in the first group according to the exponential transformation.

[0118] Among them, the quantization module 204 can introduce formula (4) to perform exponential transformation on the weights, so that the distribution of the weights becomes non-uniform, and then use formula (5) to quantize the transformed weights f(W), and after quantization, the exponential transformation is backed off to ensure equivalence before and after the transformation. The exponent α used for the transformation can be obtained by searching, and the range of α is between 0 and 1. The α finally selected is the α that minimizes the weight quantization error, as shown in formula (6):

[0119] f(W)=W α (4)

[0120] W Q =f -1 (Q[f(W)]) (5)

[0121] E(α l )=‖W l -Q -1 (f -1 (Q[f(W l )]))‖ p (6)

[0122] The Q in the above formula represents quantization, and W Q is the quantized weight value after back-off exponential transformation, W l Represents the weight value of the lth layer, p is the order of the norm, for example, p takes a value of 2 to represent the L2 norm, and E represents the weight quantization error.

[0123] Considering that there are also a small number of outliers in the weights of the AI ​​model, the use of non-uniform quantization can avoid wasting precision bits, improve accuracy, and achieve low-bit quantization. In addition, this application uses exponential transformation to achieve non-uniform quantization, improving quantization efficiency.

[0124] Based on the aforementioned data quantization system 200, the present application further provides a data quantization method, which is described below in conjunction with embodiments.

[0125] Referring to the flowchart of a data quantization method shown in FIG6 , the method includes the following steps:

[0126] S602: The data quantification system 200 runs the AI ​​model and obtains data generated during the operation of the AI ​​model.

[0127] The data generated during AI model operation can be data generated during AI model inference, such as the AI ​​model's activation value. In some examples, the data generated during AI model operation can also include AI model weights, such as weights iteratively updated during AI model training. Activation values ​​and weights are typically floating-point data. To improve inference performance, floating-point data can be quantized.

[0128] To improve the quantization effect, the data quantization system 200 can perform multiple inference operations on the AI ​​model, perform statistical analysis on the data generated by the multiple inference operations, and obtain the data generated during the operation of the AI ​​model. Taking the activation value as an example, the data quantization system 200 can perform m inference operations on the AI ​​model to obtain the activation values ​​generated by the m inference operations. The data quantization system 200 can then calculate the average value of the m activation values, or determine the maximum or minimum value. The data quantization system 200 can determine the average value, maximum value, or minimum value of the activation values ​​generated by the m inference operations as the data generated during the operation of the AI ​​model, which is used to determine the quantization parameters.

[0129] The data generated during the operation of the AI ​​model may include multiple groups. Taking the first group among the multiple groups as an example, the data in the first group may be the activation values ​​generated by at least one layer in the AI ​​model during the inference process. Alternatively, the data in the first group may be the activation values ​​generated by the same set of input data at different layers in the AI ​​model inference process, such as the activation values ​​generated by the input data of one channel at different layers in the AI ​​model inference process. For example, in an image processing scenario, the input image of the AI ​​model may include three channels, R, G, and B, and the first group may be the activation values ​​generated by the input image of the R channel at different layers in the AI ​​model inference process. Similarly, the data in the first group may be the weight of at least one layer in the AI ​​model, or the weight of processing the input data of at least one channel.

[0130] Specifically, the data quantization system 200 may provide an interface through which the AI ​​model to be quantized is obtained. The interface may be an application programming interface (API) or a user interface, such as a graphical user interface (GUI) or a command user interface (CUI).

[0131] In some possible implementations, the model training tool can call an API provided by the data quantization system 200 to provide the data quantization system 200 with the AI ​​model to be quantized. For example, the API interface parameters may include the path or storage address of the AI ​​model to be quantized. Based on the interface parameters, the data quantization system 200 can obtain the parameters of the AI ​​model to be quantized, such as the structure and weights of the AI ​​model, from the corresponding path or storage address. The data quantization system 200 inputs the input data into the AI ​​model for layer-by-layer reasoning to obtain activation values.

[0132] In some other possible implementations, the data quantization system 200 can present a quantization interface to the user. As shown in Figure 7, the quantization interface 700 can be a GUI, which carries a path input box 702 and an upload control 704. The user can enter the path of the AI ​​model to be quantized in the path input box, and then click the upload control 704 to trigger the upload operation. Accordingly, the data quantization system 200 can respond to the upload operation triggered by the user and obtain the AI ​​model to be quantized. In some examples, the quantization interface 700 can also include a browsing control 706. The user can select the AI ​​model to be quantized through the browsing control, thereby entering the address of the AI ​​model into the above-mentioned path input box 702, without the need for manual input by the user, thereby improving convenience. Among them, the data quantization system 200 can also input input data into the AI ​​model, run the AI ​​model for layer-by-layer reasoning, and obtain the activation value of each layer.

[0133] S604 : The data quantification system 200 determines the abnormality level of data in a first group among the multiple groups.

[0134] The AI ​​abnormality level indicates the degree of difference in the size of the data in the first group. The degree of difference can be an absolute degree of difference, for example, the difference between the maximum and minimum values ​​of the data in the first group. Alternatively, the degree of difference can also be a relative degree of difference, for example, the ratio of the maximum absolute value of the data in the first group to the standard deviation. Taking the activation value as an example, the abnormality level of the activation value in the first group can be the ratio of the maximum absolute value of the activation value in the first group to the standard deviation. Based on this, the data quantization system 200 can first perform a statistical analysis based on the activation values ​​in the first group to determine the maximum absolute value of the activation values ​​in the first group and the standard deviation of the activation values, and then determine the abnormality level of the activation values ​​in the first group based on the above-mentioned maximum absolute value and standard deviation.

[0135] It should be noted that using the ratio of the maximum absolute value to the standard deviation as the degree of abnormality is merely an exemplary embodiment of the present application. In other possible implementations of the embodiments of the present application, the degree of abnormality may also be other indicators and determined by other means. For example, the degree of abnormality may also be the ratio of the maximum absolute value to the variance. For another example, the degree of abnormality may also be the sum of the ratio of the maximum absolute value to the standard deviation and a set value.

[0136] S606: The data quantization system 200 determines parameters for suppressing abnormalities in the data in the first group according to the degree of abnormality.

[0137] Considering that the degree of abnormality of data in different groups may be different, if the same abnormality suppression parameters are used for scaling, for example, the activation values ​​in multiple groups are uniformly scaled using the same scaling factor, it may lead to insufficient or excessive suppression. Therefore, the data quantization system 200 can determine the scaling factor of the data in the first group based on the degree of abnormality of the data in the first group.

[0138] The data quantization system 200 can model the scaling factor and the degree of abnormality to obtain a mapping relationship between the scaling factor and the degree of abnormality. Accordingly, when determining the scaling factor, the data quantization system 200 can determine the scaling factor for the data in the first group based on the degree of abnormality of the data in the first group using the mapping relationship between the scaling factor and the degree of abnormality.

[0139] The mapping relationship between the scaling factor and the degree of anomaly can be represented by a function, the output of which is the scaling factor. The function can be a piecewise function related to the degree of anomaly. When the range of the degree of anomaly is different, the scaling factor Scale can be calculated in different ways to adapt to data quantization under different data distributions.

[0140] The degree of anomaly can be categorized based on its relationship to a threshold. In some possible implementations, the degree of anomaly can be categorized into the following situations based on its relationship to a first threshold: less than the first threshold; greater than or equal to the first threshold. When the degree of anomaly of the data in the first group is less than the first threshold, indicating that the degree of anomaly in the first group is relatively low, it can be considered that no outliers exist or that the impact of outliers is relatively small. The data quantization system 200 can determine a scaling factor for the data in the first group to be a first value. The first value can be a fixed value, such as 1, which does not suppress the data in the first group. When the degree of anomaly of the data in the first group is greater than or equal to the first threshold, the data quantization system 200 can determine a scaling factor for the activation values ​​in the first group based on the activation values ​​and weights in the first group. The weights include the weights of the network layer or channel to which the activation values ​​in the first group belong. For example, if the first group includes activation values ​​in the i-th network layer, the weights include the weights in the i-th network layer.

[0141] Furthermore, the case where the degree of abnormality is greater than or equal to the first threshold can be further divided into: greater than or equal to the first threshold and less than or equal to the second threshold; greater than the second threshold. When the degree of abnormality of the data in the first group is greater than or equal to the first threshold and less than or equal to the second threshold, indicating that the abnormality range is in the middle, the data quantization system 200 can directly balance the distribution of activation values ​​and weights to determine the scaling factor. Specifically, the data quantization system 200 can determine the scaling factor of the activation values ​​in the first group based on the ratio of the maximum activation value to the maximum weight in the first group. When the degree of abnormality of the data in the first group is greater than the second threshold, indicating that the abnormality is relatively large, the data quantization system 200 can introduce a transfer coefficient, also known as an α coefficient, to determine the scaling factor. Specifically, the data quantization system 200 can search for a transfer coefficient that satisfies the quantization error of the data in the first group. The data quantization system 200 can set a range for the α coefficient, for example, a range of 0-1 (including the endpoints), and then use a fine-grained search method to solve for the optimal α coefficient to ensure that the activation value quantization error is minimized. The scaling factor of the activation values ​​in the first group is then determined based on the maximum activation value, maximum weight, and the solved α coefficient in the first group.

[0142] The specific implementation of the data quantization system 200 determining the scaling factor according to the α coefficient, the maximum activation value, and the maximum weight can be found in the description of formula (3), which will not be repeated here.

[0143] In addition, the parameters for abnormal suppression may also include an offset factor. Taking the activation value as an example, the offset factor may be the average value of the activation values ​​in the first group, such as the arithmetic mean of the maximum activation value and the minimum activation value. For specific implementation, please refer to the description of formula (1).

[0144] S608 : The data quantization system 200 suppresses the data in the first group according to the abnormality suppression parameters.

[0145] Data in the same group can share parameters for anomaly suppression. For example, activation values ​​in the same group can share a scaling factor. The data quantization system 200 can use the scaling factor of the data in the first group to scale the data in the first group (e.g., activation values) to obtain the suppressed first group. Furthermore, to achieve an equivalent transformation, the data quantization system 200 can perform an inverse scaling process on the corresponding weights. For example, after scaling the activation values, the data quantization system 200 can divide the weights by the scaling factor to achieve an equivalent transformation.

[0146] Furthermore, the scaling factors of different groups can be represented by vectors, where each element of the vector represents the scaling factor of a group. Accordingly, the data quantization system 200 can perform scaling processing on the data in different groups through matrix multiplication operations, thereby improving efficiency.

[0147] In some possible implementations, the data quantization system 200 may also determine an offset factor based on the maximum and minimum activation values ​​in the group, and offset the activation values ​​based on the offset factor, thereby migrating the activation values ​​in the group to a range symmetrically centered around 0, thereby minimizing the range of activation values. The data quantization system 200 then performs the aforementioned scaling process on the offset activation values ​​to suppress outliers.

[0148] S610 : The data quantization system 200 quantizes the suppressed data in the first group.

[0149] Specifically, the data quantization system 200 can determine a granularity for data quantization and quantize the data in the suppressed first group according to the granularity. In some examples, the granularity of data quantization is the same as the granularity of outlier suppression. For example, the granularity of data quantization can be group-by-group quantization, specifically quantization in units of activation value groups, with the activation value groups sharing quantization parameters.

[0150] Taking into account the situation that outlier suppression cannot completely eliminate the impact of outliers, the data quantization system 200 can also perform data rollback during the calibration phase, for example, rolling back the activation values ​​of some groups to the activation values ​​before outlier suppression, to ensure the quantization accuracy of the AI ​​model.

[0151] Specifically, the data in the first group can be the activation value generated by at least one layer of the AI ​​model during the inference process. Considering that the range of activation values ​​of some network layers is very large, in order to maximize the quantization accuracy of the AI ​​model, this application introduces activation value fallback. Specifically, the data quantization system 200 can sort the multiple groups according to the degree of abnormality of the data in each group in the multiple groups, for example, sorting them in order from high to low in terms of abnormality. According to the sorting results of the multiple groups, the data in the target group in the multiple groups is restored to the data before suppression, for example, to the activation value before suppression. Among them, the target group can be the top n groups in the sorting results, or the group with an abnormality greater than a third threshold. n and the third threshold can be set based on experience, for example, n takes a value of 1 to 5, which can achieve lossless accuracy. In some examples, the data quantization system 200 can save the activation value before suppression, and when fallback is needed, the activation value before suppression can be read. In other examples, the data quantization system 200 can perform inverse processing on the activation value for outlier suppression. For example, when performing activation value rollback on the first activation value group, if the scaling factor of the activation values ​​in the first group is 0.2, the suppressed activation values ​​in the first group can be divided by the above scaling factor (equivalent to multiplying by the inverse of the scaling factor, for example, multiplying by 5) to achieve activation value rollback.

[0152] Furthermore, the present application supports multiple quantization algorithms, including but not limited to minimum-maximum quantization and histogram quantization. Different quantization algorithms are suitable for data with different data distributions. Therefore, the data quantization system 200 can adaptively select a quantization algorithm based on the characteristics of the data distribution in the group. When the abnormality of the data in the first group is less than the fourth threshold, the data quantization system 200 can perform histogram quantization on the data in the suppressed first group. Specifically, the data quantization system 200 can discard a first proportion of abnormal values ​​in the suppressed first group and quantize according to the maximum and minimum values ​​of the remaining data. When the abnormality of the data in the first group is greater than or equal to the fourth threshold, the data quantization system 200 can perform minimum-maximum quantization on the data in the suppressed first group. Specifically, the data quantization system 200 can quantize according to the maximum and minimum values ​​of the data in the suppressed first group.

[0153] Still using the activation values ​​in the first group as an example, specifically, when the degree of abnormality of the activation values ​​in the first group is less than the fourth threshold, indicating that the degree of abnormality of the activation values ​​in the first group is relatively small, the data quantization system 200 can perform histogram quantization on the suppressed activation values ​​in the first group. When the degree of abnormality of the activation values ​​in the first group is greater than or equal to the fourth threshold, indicating that the degree of abnormality of the activation values ​​in the first group is relatively large, the data quantization system 200 can perform mini-max quantization on the suppressed activation values ​​in the first group.

[0154] During the layer-by-layer quantization calibration process, the activation values ​​of all preceding layers of each layer can also use random probability to determine whether to quantize. Specifically, for the first group, the data quantization system 200 can obtain the random probability of the first group. When the random probability is greater than a second value, the suppressed data in the first group can be quantized. Compared to non-quantization, random quantization is more consistent with real-world scenarios, and compared to full quantization, random quantization can also avoid overfitting. Therefore, random quantization has better generalization accuracy.

[0155] The activation value quantization process is described above. The data quantization system 200 can also quantize weights. The weight quantization is described in detail below.

[0156] Considering that the weight distribution of the AI ​​model is uneven and there is a certain proportion of outliers, if uniform quantization is used, a large amount of precision bits may be wasted due to the lack of adaptive partitioning capability of the quantization resolution. The data quantization system 200 can use a non-uniform quantization mechanism to quantize the weights.

[0157] Among them, non-uniform quantization can be achieved by introducing an exponential function to equivalently transform the quantization formula to obtain non-uniform and adaptive quantization resolution, which can more friendly quantize weights containing outliers. Specifically, the data quantization system 200 can perform an exponential transformation on the weights in the suppressed first group to obtain transformed weights. The data quantization system 200 can then quantize the transformed weights and perform an inverse transformation of the exponential transformation on the quantized weights to ensure equivalent transformation.

[0158] When performing exponential transformation, the data quantization system 200 may first determine an exponent. Taking the first group as an example, the data quantization system 200 may search for an exponent that satisfies the weight quantization error condition, and then perform exponential transformation on the weights in the first group based on the exponent.

[0159] The process of determining the exponent, performing exponential transformation according to the exponent, and performing quantization based on the exponential transformation result can refer to the embodiment shown in FIG2 , for example, the description of the relevant contents of formulas (4) to (6) in the embodiment shown in FIG2 , which will not be repeated here.

[0160] Based on the above description, it can be seen that the data quantization method of the present application takes into account the wide distribution of the amplitude range of outliers in the data of different network layers or different channels in the AI ​​model. Using a single suppression method may result in insufficient or excessive suppression. It is proposed to suppress outliers based on the data distribution of different groups. For different groups, the degree of abnormality is determined according to their respective data distributions, and the parameters for abnormal suppression of the data in the group are determined according to the degree of abnormality. This can avoid insufficient or excessive suppression, and the quantization accuracy of the AI ​​model can be improved by adaptively suppressing different groups. Moreover, in the process of quantizing the activation value, the quantization accuracy of the AI ​​model can be guaranteed and the generalization ability can be improved by activation value fallback, adaptive calibration or random quantization of the activation value. In addition, for weight quantization, the present application introduces exponential transformation to achieve high-precision low-bit non-uniform quantization to reduce precision bit waste.

[0161] Based on the aforementioned data quantization method, the present application further provides a data quantization system 200. As shown in FIG2 , the data quantization system 200 includes:

[0162] An outlier suppression module 202 is configured to run an AI model, obtain data generated during the running of the AI ​​model, the data including multiple groups, determine a degree of abnormality of data in a first group of the multiple groups, the degree of abnormality indicating a degree of difference in size of the data in the first group, determine parameters for performing abnormality suppression on the data in the first group based on the degree of abnormality, and suppress the data in the first group based on the abnormality suppression parameters;

[0163] The quantization module 204 is configured to quantize the data in the suppressed first group.

[0164] The above-mentioned outlier suppression module 202 is used to implement the contents described in S602 to S608 in the embodiment of Figure 6, and the quantization module 204 is used to implement the contents described in S610. The outlier suppression module 202 and the quantization module 204 can be implemented by hardware or software.

[0165] When implemented via software, the outlier suppression module 202 and the quantification module 204 can be applications running on a computing device, such as a computing engine. These applications can also be virtualized and provided to users through virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. A VM service can be a service that uses virtualization technology to create a virtual machine (VM) resource pool on multiple physical hosts to provide users with VMs on demand. A BMS service is a service that creates a BMS resource pool on multiple physical hosts to provide users with BMSs on demand. A container service is a service that creates a container resource pool on multiple physical hosts to provide users with containers on demand. A VM is a simulated virtual computer, or a logical computer. BMS is a scalable, high-performance computing service with computing performance comparable to that of a traditional physical machine and secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are only specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0166] When implemented through hardware, the outlier suppression module 202 and the quantization module 204 may include at least one computing device, such as a server. Alternatively, the outlier suppression module 202 and the quantization module 204 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0167] In some possible implementations, the data generated during the operation is the activation value generated by each layer of the AI ​​model during at least one inference process.

[0168] In some possible implementations, the data in the first group are activation values ​​generated by at least one layer in the AI ​​model during the inference process, or are activation values ​​generated by different layers in the inference process of the AI ​​model for the same set of input data.

[0169] In some possible implementations, the parameter for anomaly suppression is a scaling factor, and the outlier suppression module 202 is specifically configured to:

[0170] When the abnormality level is less than a first threshold, determining a scaling factor of the data in the first group to be a first value;

[0171] When the abnormality level is greater than or equal to the first threshold, a scaling factor of the activation values ​​in the first group is determined according to the activation values ​​and weights in the first group.

[0172] In some possible implementations, the outlier suppression module 202 is specifically configured to:

[0173] When the abnormality level is less than or equal to a second threshold, determining a scaling factor of the activation value in the first group according to a ratio of a maximum activation value to a maximum weight in the first group;

[0174] When the degree of abnormality is greater than the second threshold, a transfer coefficient is searched so that the quantization error of the data in the first group meets the condition, and a scaling factor of the activation value in the first group is determined according to the maximum activation value, the maximum weight and the transfer coefficient in the first group.

[0175] In some possible implementations, the system 200 further includes:

[0176] The calibration module 206 is used to sort the multiple groups according to the abnormality level of the data in each group of the multiple groups, and restore the data in the target group of the multiple groups to the data before suppression based on the sorting results of the multiple groups, where the target group is a group with an abnormality level greater than a third threshold, or a group ranked in the top n in terms of abnormality level from high to low.

[0177] Similar to the outlier suppression module 202 and the quantization module 204, the calibration module 206 can be implemented in software or hardware. When implemented in software, the calibration module 206 can be an application running on a computer device, such as a VM service, a BMS service, or a container service. When implemented in hardware, the calibration module 206 can include at least one computing device, such as a server. Alternatively, the calibration module 206 can be implemented using an ASIC or a PLD.

[0178] In some possible implementations, the quantization module 204 is specifically configured to:

[0179] When the abnormality degree of the data in the first group is less than a fourth threshold, discarding a first proportion of abnormal values ​​from the suppressed data in the first group, and quantizing the remaining data according to the maximum and minimum values;

[0180] When the abnormality degree of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the suppressed data in the first group.

[0181] In some possible implementations, the quantization module 204 is specifically configured to:

[0182] Obtaining a random probability of the first group;

[0183] When the random probability is greater than a second value, quantizing the data in the suppressed first group.

[0184] In some possible implementations, the data in the first group includes activation values ​​and weights in the first group, and the quantization module 204 is specifically configured to:

[0185] Performing exponential transformation on the weights in the suppressed first group to obtain transformed weights;

[0186] The transformed weights are quantized, and an inverse transform of the exponential transform is performed on the quantized weights.

[0187] In some possible implementations, the quantization module 204 is specifically configured to:

[0188] Searching for an index that satisfies a condition for the quantization error of the weights in the first group after suppression;

[0189] The weights in the suppressed first group are exponentially transformed according to the exponent.

[0190] This application also provides a computing device 800. As shown in Figure 8, computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. Processor 804, memory 806, and communication interface 808 communicate with each other via bus 802. Computing device 800 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 800.

[0191] Bus 802 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 802 may include a path for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, and communication interface 808).

[0192] The processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0193] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD). Memory 806 stores executable program code, and processor 804 executes the executable program code to implement the aforementioned data quantization method. Specifically, memory 806 stores instructions for data quantization system 200 to execute the data quantization method.

[0194] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0195] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0196] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store instructions for executing the data quantization method using the same data quantization system 200.

[0197] In some possible implementations, one or more computing devices 800 in the computing device cluster may also be used to execute some of the instructions of the data quantization system 200 for executing the data quantization method. In other words, a combination of one or more computing devices 800 may jointly execute the instructions of the data quantization system 200 for executing the data quantization method.

[0198] It should be noted that the memories 806 in different computing devices 800 in the computing device cluster may store different instructions for executing partial functions of the data quantization system 200 .

[0199] FIG10 illustrates a possible implementation. As shown in FIG10 , two computing devices 800A and 800B are connected via a communication interface 808. The memory in computing device 800A stores instructions for executing the functions of outlier suppression module 202. The memory in computing device 800B stores instructions for executing the functions of quantization module 204. Furthermore, the memory in computing device 800A also stores instructions for executing the functions of calibration module 206. In other words, the memories 806 of computing devices 800A and 800B collectively store instructions for data quantization system 200 to execute the data quantization method.

[0200] The connection mode between the computing device clusters shown in FIG10 may be considered to be based on the fact that the data quantization method provided in this application requires more computing power for quantization. Therefore, it is considered to delegate the functions implemented by the quantization module 204 to the computing device 800B.

[0201] It should be understood that the functionality of the computing device 800A shown in FIG10 may also be implemented by multiple computing devices 800. Similarly, the functionality of the computing device 800B may also be implemented by multiple computing devices 800.

[0202] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG11 shows a possible implementation. As shown in FIG11 , two computing devices 800C and 800D are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800C stores instructions for executing the functions of the outlier suppression module 202. At the same time, the memory 806 in the computing device 800D stores instructions for executing the functions of the quantization module 204. Furthermore, the memory in the computing device 800C also stores instructions for executing the functions of the calibration module 206.

[0203] The connection method between the computing device clusters shown in Figure 11 can be considered to be that the data quantization method provided in this application requires more computing power for quantization, so the functions implemented by the quantization module 204 are considered to be handed over to the computing device 800D for execution.

[0204] It should be understood that the functionality of the computing device 800C shown in FIG11 may also be accomplished by multiple computing devices 800. Similarly, the functionality of the computing device 800D may also be accomplished by multiple computing devices 800.

[0205] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data quantization method described above for use in the data quantization system 200.

[0206] Embodiments of the present application also provide a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be executed on a computing device or stored on any usable medium. When the computer program product is executed on at least one computing device, the at least one computing device executes the aforementioned data quantization method.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data quantization method, characterized in that: The method comprises: Running an artificial intelligence (AI) model and acquiring data generated during the running of the AI ​​model, wherein the data includes multiple groups; determining a degree of abnormality of data in a first group of the plurality of groups, the degree of abnormality indicating a degree of difference in size of the data in the first group; determining parameters for suppressing abnormality of the data in the first group according to the abnormality degree; Suppressing the data in the first group according to the abnormality suppression parameters; The data in the first cohort after inhibition were quantified.

2. The method according to claim 1, characterized in that The data generated during the operation is the activation value generated by each layer of the AI ​​model during at least one inference process.

3. The method according to claim 1, characterized in that The data in the first group are activation values ​​generated by at least one layer in the AI ​​model during the inference process, or are activation values ​​generated by different layers in the inference process of the AI ​​model for the same set of input data.

4. The method according to any one of claims 1 to 3, characterized in that The parameter for suppressing anomalies is a scaling factor, and determining the parameter for suppressing anomalies for the data in the first group according to the degree of anomaly includes: When the abnormality level is less than a first threshold, determining a scaling factor of the data in the first group to be a first value; When the abnormality level is greater than or equal to the first threshold, a scaling factor of the activation values ​​in the first group is determined according to the activation values ​​and weights in the first group.

5. The method according to claim 4, characterized in that The determining a scaling factor of the activation values ​​in the first group according to the activation values ​​and weights in the first group includes: When the abnormality level is less than or equal to a second threshold, determining a scaling factor of the activation value in the first group according to a ratio of a maximum activation value to a maximum weight in the first group; When the degree of abnormality is greater than the second threshold, a transfer coefficient is searched so that the quantization error of the data in the first group meets the condition, and a scaling factor of the activation value in the first group is determined according to the maximum activation value, the maximum weight and the transfer coefficient in the first group.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: sorting the plurality of groups according to a degree of abnormality of the data in each of the plurality of groups; According to the sorting results of the multiple groups, data in a target group among the multiple groups is restored to data before suppression, wherein the target group is a group whose abnormality is greater than a third threshold, or a group ranked in the top n in terms of abnormality from high to low.

7. The method according to any one of claims 1 to 6, characterized in that The quantifying the data in the suppressed first group includes: When the abnormality degree of the data in the first group is less than a fourth threshold, discarding a first proportion of abnormal values ​​from the suppressed data in the first group, and quantizing the remaining data according to the maximum and minimum values; When the abnormality degree of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the suppressed data in the first group.

8. The method according to any one of claims 1 to 7, characterized in that The quantifying the data in the suppressed first group includes: Obtaining a random probability of the first group; When the random probability is greater than a second value, quantizing the data in the suppressed first group.

9. The method according to any one of claims 1 to 8, characterized in that The data in the first group includes activation values ​​and weights in the first group, and quantizing the suppressed data in the first group includes: Performing exponential transformation on the weights in the suppressed first group to obtain transformed weights; The transformed weights are quantized, and an inverse transform of the exponential transform is performed on the quantized weights.

10. The method according to claim 9, characterized in that The step of performing an exponential transformation on the weights in the suppressed first group to obtain the transformed weights includes: Searching for an index that satisfies a condition for the quantization error of the weights in the first group after suppression; The weights in the suppressed first group are exponentially transformed according to the exponent.

11. A data quantization system, characterized in that: The system comprises: an outlier suppression module, configured to run an artificial intelligence (AI) model, obtain data generated during the running of the AI ​​model, the data including a plurality of groups, determine a degree of abnormality of data in a first group of the plurality of groups, the degree of abnormality indicating a degree of difference in size of the data in the first group, determine parameters for performing abnormality suppression on the data in the first group based on the degree of abnormality, and suppress the data in the first group based on the abnormality suppression parameters; The quantization module is used to quantize the data in the suppressed first group.

12. The system according to claim 11, wherein: The data generated during the operation is the activation value generated by each layer of the AI ​​model during at least one inference process.

13. The system according to claim 11, wherein: The data in the first group are activation values ​​generated by at least one layer in the AI ​​model during the inference process, or are activation values ​​generated by different layers in the inference process of the AI ​​model for the same set of input data.

14. The system according to any one of claims 11 to 13, characterized in that: The parameter of the anomaly suppression is a scaling factor, and the anomaly suppression module is specifically used to: When the abnormality level is less than a first threshold, determining a scaling factor of the data in the first group to be a first value; When the abnormality level is greater than or equal to the first threshold, a scaling factor of the activation values ​​in the first group is determined according to the activation values ​​and weights in the first group.

15. The system according to claim 14, wherein: The outlier suppression module is specifically used for: When the abnormality level is less than or equal to a second threshold, determining a scaling factor of the activation value in the first group according to a ratio of a maximum activation value to a maximum weight in the first group; When the degree of abnormality is greater than the second threshold, a transfer coefficient is searched so that the quantization error of the data in the first group meets the condition, and a scaling factor of the activation value in the first group is determined according to the maximum activation value, the maximum weight and the transfer coefficient in the first group.

16. The system according to any one of claims 11 to 15, characterized in that The system further comprises: A calibration module is configured to sort the multiple groups according to the degree of abnormality of the data in each of the multiple groups, and restore the data in a target group among the multiple groups to the data before suppression based on the sorting results of the multiple groups, where the target group is a group whose abnormality is greater than a third threshold, or a group ranked in the top n in descending order of abnormality.

17. The system according to any one of claims 11 to 16, characterized in that The quantization module is specifically used for: When the abnormality degree of the data in the first group is less than a fourth threshold, discarding a first proportion of abnormal values ​​from the suppressed data in the first group, and quantizing the remaining data according to the maximum and minimum values; When the abnormality degree of the data in the first group is greater than or equal to the fourth threshold, quantization is performed based on the maximum and minimum values ​​of the suppressed data in the first group.

18. The system according to any one of claims 11 to 17, characterized in that The quantization module is specifically used for: Obtaining a random probability of the first group; When the random probability is greater than a second value, quantizing the data in the suppressed first group.

19. The system according to any one of claims 11 to 18, characterized in that The data in the first group includes activation values ​​and weights in the first group, and the quantization module is specifically configured to: Performing exponential transformation on the weights in the suppressed first group to obtain transformed weights; The transformed weights are quantized, and an inverse transform of the exponential transform is performed on the quantized weights.

20. The system according to claim 19, wherein: The quantization module is specifically used for: Searching for an index that satisfies a condition for the quantization error of the weights in the first group after suppression; The weights in the suppressed first group are exponentially transformed according to the exponent.

21. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the data quantization method according to any one of claims 1 to 10.

Images