Data processing method, electronic device, and storage medium

Abstract

A data processing method, an electronic device, and a storage medium. The data processing method includes: performing quantitative reasoning on a neural network, determining the outlier ratio of each operator among multiple operators included in the neural network, wherein the outlier ratio of an operator is the proportion of outliers in an output activation tensor output by the operator during the quantitative reasoning process; adjusting a first quantization scheme according to the outlier ratio of each operator, wherein in the first quantization scheme, the quantization accuracy of at least one of the weight parameters and activation tensors of the neural network is lower than the original accuracy; and performing reasoning operations on the neural network according to the adjusted first quantization scheme. The data processing method uses outlier statistics of the output activation tensor output by each operator to determine the quantization accuracy of the operator, replacing the method of judging the quantization accuracy of the operator by manual experience, so as to maintain low-bit quantization performance while improving the accuracy of model quantization reasoning.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Large Language Models (LLMs) are neural network models with a large number of parameters based on deep learning technology, designed to understand and generate natural language. Large language models usually have the following characteristics: the model has a large number of parameters, usually between millions and billions, which enables them to learn rich language features and patterns; based on deep neural networks, especially models based on the Transformer architecture, these networks are large in scale and complex in structure, and can capture the complexity and diversity of language. Large language models play an important role in the field of natural language processing (NLP). They can generate natural language text, perform machine translation, sentiment analysis, question-answering systems, and many other tasks. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a data processing method, including: performing quantized inference on a neural network, determining the outlier ratio of each operator among multiple operators included in the neural network, wherein the outlier ratio of the operator is the proportion of outliers in an output activation tensor output by the operator during the quantized inference process; adjusting a first quantization scheme according to the outlier ratio of each operator, wherein, in the first quantization scheme, the quantization accuracy of at least one of the weight parameters and activation tensors of the neural network is lower than the original accuracy; and performing inference operations on the neural network according to the adjusted first quantization scheme.

[0004] For example, in the data processing method provided by at least one embodiment of the present disclosure, the multiple operators include a first operator and a second operator, and the adjusted first quantization scheme stipulates that when performing the inference operation, the quantization accuracy of at least one of the activation tensors and weight parameters of the first operator and the second operator is different.

[0005] For example, in the data processing method provided by at least one embodiment of the present disclosure, the outlier ratio of the first operator is greater than or equal to a preset threshold, the outlier ratio of the second operator is less than the preset threshold, the quantization accuracy of the activation tensor of the first operator is the original accuracy, and the quantization accuracy of the activation tensor of the second operator is lower than or equal to the original accuracy.

[0006] For example, in the data processing method provided by at least one embodiment of the present disclosure, the quantitative reasoning includes first quantitative reasoning and second quantitative reasoning, the outlier ratio includes first outlier ratio and second outlier ratio, and the neural network is subjected to quantitative reasoning to determine the outlier ratio of each of the multiple operators included in the neural network, including: performing the first quantitative reasoning on the neural network using the first quantization scheme, and determining the first outlier ratio of each operator according to the result of the first quantitative reasoning; performing the second quantitative reasoning on the neural network using the second quantization scheme, and determining the second outlier ratio of each operator according to the result of the second quantitative reasoning; wherein the first quantization scheme is different from the second quantization scheme, and the second quantization scheme stipulates that the second quantitative reasoning is performed according to the original precision of the weight parameters of the neural network and the activation tensor of the neural network.

[0007] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first quantization scheme is used to perform the first quantization inference on the neural network, and the first outlier ratio of each operator is determined according to the result of the first quantization inference, including: performing the first quantization inference according to the quantization scheme of the weight parameters of the neural network and the activation tensor of the neural network specified by the first quantization scheme to obtain the output activation tensor of each operator, wherein the multiple operators all use the same quantization scheme when performing the first quantization inference; for each operator, determining the number of outliers in the output activation tensor, and determining the ratio of the number to the total number of elements in the output activation tensor as the first outlier ratio.

[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, a second quantization scheme is used to perform second quantization inference on the neural network, and a second outlier ratio of each operator is determined according to the result of the second quantization inference, including: performing the second quantization inference according to the weight parameters of the neural network and the original precision of the activation tensor to obtain the output activation tensor of each operator; for each operator, determining the number of outliers in the output activation tensor, and determining the ratio of the number to the total number of elements in the output activation tensor as the second outlier ratio.

[0009] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first quantization scheme is adjusted according to the outlier ratio of each operator, including: classifying the operator into a first type of operator or a second type of operator according to a comparison result between the outlier ratio of the operator and a preset threshold, wherein the outlier ratio of the first type of operator is greater than or equal to the preset threshold; based on the first quantization scheme, adjusting the quantization scheme of the first type of operator, maintaining the quantization scheme of the second type of operator, to obtain the adjusted first quantization scheme, wherein, in the adjusted first quantization scheme, the quantization accuracy of the activation tensor of the first type of operator is higher than or equal to the quantization accuracy of the activation tensor of the second type of operator.

[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, in response to the quantitative reasoning including the first quantitative reasoning and the second quantitative reasoning, the outlier ratio includes the first outlier ratio and the second outlier ratio, and according to the comparison result of the outlier ratio of the operator and the preset threshold, the operator is classified into the first type of operator or the second type of operator, including: in response to the first outlier ratio and the second outlier ratio of the operator being higher than the preset threshold, determining that the operator is the first type of operator; in response to at least one of the first outlier ratio and the second outlier ratio of the operator being lower than the preset threshold, determining that the operator is the second type of operator.

[0011] For example, in the data processing method provided by at least one embodiment of the present disclosure, in the adjusted first quantization scheme, the quantization scheme of the first type of operator stipulates that the activation tensor of the first type of operator is quantized to the original precision to perform the inference operation.

[0012] For example, in the data processing method provided by at least one embodiment of the present disclosure, in the adjusted first quantization scheme, the quantization scheme of the first type of operator stipulates that the weight parameters of the first type of operator are quantized into 8-bit integers for performing the inference operation.

[0013] For example, in the data processing method provided in at least one embodiment of the present disclosure, in response to the electronic device executing the data processing method supporting calculations in the FP8 data format, in the adjusted first quantization scheme, the quantization scheme of the first type of operator stipulates that the weight parameters of the first type of operator are quantized to FP8 for performing the inference operation.

[0014] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: optimizing the network graph of the neural network based on the adjusted first quantization scheme.

[0015] For example, in the data processing method provided in at least one embodiment of the present disclosure, the network graph of the neural network is optimized based on the adjusted first quantization scheme, including: for multiple operators with contextual connections, according to the adjusted first quantization scheme, sub-graphs of operators with consistent input and output precision are fused.

[0016] For example, in the data processing method provided in at least one embodiment of the present disclosure, the neural network is a large language class model.

[0017] At least one embodiment of the present disclosure provides a data processing device, including: a quantization reasoning training module, configured to perform quantization reasoning on a neural network, and determine the outlier ratio of each operator among multiple operators included in the neural network, wherein the outlier ratio of the operator is the proportion of outliers in an output activation tensor output by the operator during the quantization reasoning process; an adjustment module, configured to adjust a first quantization scheme according to the outlier ratio of each operator, wherein, in the first quantization scheme, the quantization accuracy of at least one of the weight parameters and activation tensors of the neural network is lower than the original accuracy; and an inference module, configured to perform inference operations on the neural network according to the adjusted first quantization scheme.

[0018] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transitorily stores computer-executable instructions; and a processor, which is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the data processing method according to any embodiment of the present disclosure.

[0019] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0021] Figure 1 A schematic flow chart of a data processing method provided for at least one embodiment of the present disclosure;

[0022] Figure 2A Schematic diagram of quantized precision mapping for neural networks;

[0023] Figure 2B A schematic diagram of a neural network quantization accuracy mapping provided by an embodiment of the present disclosure;

[0024] Figure 3 A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure;

[0025] Figure 4 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0029] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0030] The reasoning of large language models requires a lot of video memory and computing power. In order to improve the reasoning efficiency and reduce the cost of deploying large language models in application scenarios such as AI (artificial intelligence), large language models are generally quantized for reasoning. Quantized reasoning of neural networks refers to the process of converting floating-point numbers (such as FP32) weights, activation tensors, etc. in the neural network model into low-precision floating-point numbers or integers (such as int8 / uint8) and performing reasoning calculations under this low-precision representation to reduce model size and storage requirements, increase reasoning speed, and reduce power consumption. Here, the process of converting high-precision floating-point numbers to low-precision floating-point numbers or integers is called quantization.

[0031] The quantitative reasoning of large language models mainly involves the quantization of two types of data: weight parameters (Weight) and activation tensors (Activation Tensor).

[0032] The two types of data have different characteristics. The amount of data for weight parameters is large, and the parameter scale can reach tens of Gigabytes or even hundreds of G, but the numerical range of weight parameters is relatively small. Generally, 8 bits are sufficient to represent it, and there will be no loss of precision, but for example, 6 bits will have irreversible loss of precision. The amount of data for activation tensors is not as large as that for weight parameters, but the numerical range is very large. The values ​​of some elements may reach thousands or tens of thousands, but the values ​​of most elements are usually between 0 and 1. Usually, elements in a tensor that differ from the mean of the tensor by more than three standard deviations are defined as outliers.

[0033] The main quantitative reasoning method of large language models also uses different quantization schemes for these two main types of data.

[0034] For example, in some quantization schemes, such as the W16A16 scheme, weight parameters and activation tensors use original precision, such as FP16 (Half-precision floating-point) or BF16 (BrainFloating-Point Format). This quantization inference method has no loss of precision, but poor performance and greater demand for video memory. Here, original precision refers to the highest precision without any quantization, that is, the original precision of weight parameters and activation tensors.

[0035] For example, in some quantization schemes, such as the W8A16 scheme or the W4A16 AWQ (Activation-aware Weight Quantization) scheme, the activation tensor uses FP16 or BF16, and the weight parameters use INT8 (8-bit integer) or INT4 (4-bit integer) quantization. Compared with the above method in which both the weight parameters and the activation tensor use the original precision, this method can improve the inference performance and optimize the demand for video memory occupancy, but it will cause precision loss.

[0036] For example, in some quantization schemes, such as the W8A8 INT8 SQ (SmoothQuant) scheme, both weight parameters and activation tensors are quantized using INT8. This method can improve performance but will result in a large loss of precision. This is because the value range of the activation tensor is very large, and some elements may have values ​​of thousands or tens of thousands, but even INT8 can only represent 255 at most. Using INT8 or INT4 quantization will cause many outliers in the activation tensor to be ignored and cannot be represented, resulting in precision loss. In this scheme, SmoothQuant can be used to normalize the weight parameters and activation tensors to appropriately improve the precision loss caused by outliers. SmoothQuant is an efficient post-training quantization (PTQ) method for large language models. It aims to transfer the difficulty of quantization from activation to weight through mathematically equivalent transformations, thereby achieving quantization of 8-bit weight parameters and 8-bit activation tensors while maintaining the accuracy of the model.

[0037] For example, in some quantization schemes, such as the W8A8 FP8 scheme, both the activation tensor and the weight parameter are quantized using FP8 (8-bit floating point). The FP8 numerical format is very suitable for Transformer-type large language model reasoning due to its large numerical expression range. The W8A8 FP8 quantization reasoning scheme is similar to the W8A8 INT8 SQ quantization scheme in terms of performance, but is superior to the latter in terms of reasoning accuracy. It can achieve lossless reasoning accuracy and is an ideal quantization scheme for LLM model quantization reasoning. Another example is the W4A8 FP8 scheme, in which the activation tensor is quantized using FP8 and the weight parameter is quantized using INT4. This scheme can further improve performance compared to the W8A8 FP8 and is the most extreme performance-oriented quantitative reasoning scheme. The performance is improved by more than 20% compared to the W8A8 FP8 and W8A8INT8 SQ, but there is a certain loss in accuracy, generally reaching more than 3%, which is not suitable for precision-sensitive AI application scenarios. However, both solutions are only applicable to some newer GPU models and a few AI NPU chips. Only these chips support FP8 calculations. Most current GPUs do not support the FP8 data type, so chip support is limited and the breadth of chip support is not as good as the other solutions mentioned above.

[0038] When inferring large language models, the numerical expression range of the output activation tensor of each layer of the neural network is generally larger than that of the convolutional neural network for computer vision. When the convolutional neural network for computer vision generally uses quantized inference, the output activation tensor (Activation Tensor) of each layer of the convolutional neural network is easier to linearly map to a narrower numerical range, which makes it easier for the convolutional neural network to quantize using low-bit data. The convolutional neural network uses INT8 quantization, and the loss of neural network inference accuracy can be controlled within 0.5% regardless of post-training quantization or training-aware quantization (Quantization Aware Training).

[0039] The weight parameters of large language models are relatively easy to quantize, and the extreme quantization in 4-bit format can also maintain high accuracy. However, as mentioned above, since there are many outliers inside the activation tensor and the range of numerical expression is large, the use of low-bit quantization will result in a large loss of inference accuracy. This is also the reason why the above-mentioned W8A8 INT8 SQ solution introduces a large loss of inference accuracy when using INT8 quantization for the activation tensor. W8A8 INT8 SQ uses SmoothQuant transformation to quantize the weight parameters and activation tensors of the neural network, which can reduce the numerical range of outliers in the activation tensor (Activation Tensor) to a certain extent, thereby reducing the loss of INT8 quantization inference accuracy. However, pure SmoothQuant quantization cannot obtain nearly lossless quantization inference accuracy in many popular large language models, and some large language models with high nonlinearity still have up to 3% accuracy loss.

[0040] Therefore, for the above quantization schemes, even the best performing ones such as W8A8 INT8 SQ and W4A8 FP8 have obvious precision loss, especially for the common Llama2-7b model, the inference precision loss of W8A8 INT8 SQ is more serious. For W4A8 FP8, there is a problem of chip support, and many chips currently cannot support this quantization scheme.

[0041] Quantized reasoning generally does not mean that all node operators of the entire neural network are calculated using the target quantization precision. It only quantizes some node operators with large data bandwidth requirements and high computational density. This step is called the neural network quantization precision map. Traditional quantization precision mapping generally adopts the operator whitelist method, and the network node operators that need to be calculated using high-precision numerical calculations are specified by artificial experience. For example, it is artificially specified that for activation tensors, linear operators are quantized using INT8 and nonlinear operators are quantized using FP16. However, this artificial experience specifies that some node operators in the network use high-precision numerical calculations. Obviously, it does not have good generalization. When the same quantization precision mapping as above is applied to the Llama2 large model with higher nonlinearity, it will bring very serious precision loss. For example, if the linear operators in the Llama2 large model are manually specified to be quantized using INT8 and the nonlinear operators are quantized using FP16, it will lead to a 7.6% loss of reasoning precision.

[0042] At least one embodiment of the present disclosure provides a data processing method and device, an electronic device, and a non-transient storage medium. The data processing method includes: performing quantitative reasoning on a neural network, determining the outlier ratio of each operator among multiple operators included in the neural network, wherein the outlier ratio of the operator is the proportion of outliers in the output activation tensor output by the operator during the quantitative reasoning process; adjusting the first quantization scheme according to the outlier ratio of each operator, wherein in the first quantization scheme, the quantization accuracy of at least one of the weight parameters and activation tensors of the neural network is lower than the original accuracy; performing the reasoning operation of the neural network according to the adjusted first quantization scheme.

[0043] The data processing method provided by at least one embodiment of the present disclosure adopts the outlier statistics of the output activation tensor of each operator output to determine the quantization accuracy of the operator, replacing the method of judging the quantization accuracy of the operator based on manual experience, so as to maintain low-bit quantization performance while improving the accuracy of model quantization reasoning.

[0044] The data processing method provided in the embodiment of the present disclosure can be applied to the data processing device provided in the embodiment of the present disclosure, and the data processing device can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a hardware device such as a mobile phone, a tablet computer, a laptop computer, etc.

[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0046] Figure 1 The following is a schematic flow chart of a data processing method provided by at least one embodiment of the present disclosure. Figure 1 As shown, the data processing method provided by at least one embodiment of the present disclosure includes steps S10-S30.

[0047] In step S10, quantitative reasoning is performed on the neural network to determine the outlier ratio of each operator in the multiple operators included in the neural network.

[0048] For example, operators in neural networks usually refer to the basic mathematical operations or operations used in the network layers of the model. These operators are used to build the various layers and components of the network to achieve data transmission, conversion and calculation. They are the basic building blocks of the network model, defining the structure and operation flow of the model, including input, output and intermediate calculations. In the model, the connection relationship between operators forms a directed graph, which reflects the calculation order of different operations in the model. By combining these operators, complex and powerful neural network models can be constructed to handle various complex tasks and data. For example, the following are some common neural network operators: linear layer, convolutional layer, pooling layer, recurrent layer, activation function layer, etc.

[0049] For example, the neural network is a large language model, and the present disclosure does not specifically limit the structure of the large language model. Of course, the present disclosure is not limited to this, and other models with similar characteristics to the large language model (for example, the numerical range of the activation tensor is very large) can also be used as the neural network model of the present disclosure.

[0050] For example, the outlier ratio of an operator is the ratio of outliers in the output activation tensor output by the operator during the quantized inference process. As mentioned above, elements in a tensor that differ from the mean of the tensor by more than three standard deviations are defined as outliers. Each operator outputs an output activation tensor during the quantized inference process, and the ratio of outliers in the output activation tensor to the total number of elements in the output activation tensor is taken as the outlier ratio of the operator.

[0051] In step S20, the first quantization scheme is adjusted according to the outlier ratio of each operator.

[0052] For example, in a first quantization scheme, at least one of a weight parameter and an activation tensor of a neural network has a quantization accuracy lower than an original accuracy.

[0053] For example, the first quantization scheme may be any one of the aforementioned W8A16, W4A16 AWQ, W8A8 INT8 SQ, W8A8 FP8, and W4A8 FP8. Of course, the first quantization scheme is not limited thereto, and may also be other feasible quantization schemes, which are not listed here one by one.

[0054] For example, the quantization accuracy is the low precision after the data is quantized, such as INT8, and the original accuracy is the highest precision of the data without any quantization, such as FP16 or BF16.

[0055] For example, in the first quantization scheme, the weight parameters of the neural network are quantized so that their quantization accuracy is lower than the original accuracy, for example, the quantization accuracy of the weight parameters is INT8, and the original accuracy is FP16; or, the activation tensor of the neural network is quantized so that its quantization accuracy is lower than the original accuracy, for example, the quantization accuracy of the activation tensor is INT8, and the original accuracy is FP16; or, both the activation tensor and weight parameters of the neural network are quantized so that their quantization accuracy is lower than the original accuracy, for example, the quantization accuracy of the weight parameters is INT4, the quantization accuracy of the activation tensor is FP8, and the original accuracy is FP16.

[0056] In step S30, the inference operation of the neural network is performed according to the adjusted first quantization scheme.

[0057] For example, in subsequent reasoning operations of the neural network, such as executing a formal reasoning task, quantized reasoning is performed according to the adjusted first quantization scheme.

[0058] The adjusted first quantization scheme determines the quantization precision mapping according to the outlier ratio of each operator, that is, determines which operators use high-precision quantization reasoning and which operators use low-precision quantization reasoning according to the outlier ratio of each operator.

[0059] For example, the multiple operators include a first operator and a second operator, and the adjusted first quantization scheme stipulates that when performing an inference operation, the quantization precision of at least one of the activation tensor and the weight parameter of the first operator and the second operator is different, for example, the quantization precision of the activation tensor of the first operator is different from the quantization precision of the activation tensor of the second operator, and / or the quantization precision of the weight parameter of the first operator is different from the quantization precision of the weight parameter of the second operator. Therefore, the quantization scheme provided in at least one embodiment of the present disclosure performs mixed precision inference, and some operators use high-precision quantization inference, and some operators use low-precision quantization inference.

[0060] In at least one embodiment of the present disclosure, a systematic fully automatic mixed precision reasoning is provided, which calculates the outlier ratio of each operator by statistically outputting the numerical range of the activation tensor, and automatically determines which operators of the neural network are suitable for low-precision quantization and which operators need to maintain high-precision numerical calculations based on the outlier ratio. While maintaining the performance advantages of low-bit quantization reasoning (such as W8A8 INT8 SQ, W4A8 FP8, W4A16 AWQ and other solutions), the loss of reasoning accuracy is controlled within the range of almost lossless accuracy (the target accuracy loss can be controlled within 0.5%).

[0061] For example, the outlier ratio of the first operator is greater than or equal to a preset threshold, the outlier ratio of the second operator is less than the preset threshold, and the quantization accuracy of the activation tensor of the first operator is higher than or equal to the quantization accuracy of the activation tensor of the second operator. For example, the quantization accuracy of the activation tensor of the first operator is the original accuracy, and the quantization accuracy of the activation tensor of the second operator is lower than or equal to the original accuracy.

[0062] If the outlier ratio of the operator is determined to be greater than or equal to a preset threshold, the operator is defined as a high-precision computing operator, and the quantization accuracy of its activation tensor can be, for example, the original accuracy. This can maintain the range of values ​​that the activation tensor can express, reduce and avoid the precision loss problem caused by the neglect of outliers due to quantization, thereby improving the reasoning accuracy and controlling the reasoning accuracy loss within the range of almost lossless accuracy.

[0063] If the outlier ratio of the operator is determined to be less than the preset threshold, the operator is defined as a low-precision calculation operator. The precision loss caused by the ignored outliers has little effect on the reasoning result. Therefore, the operator can be quantized using a performance-enhancing quantization method to maintain the performance advantage of low-bit quantization reasoning.

[0064] For example, in some embodiments, the quantitative reasoning includes a first quantitative reasoning and a second quantitative reasoning, and the outlier ratio includes a first outlier ratio and a second outlier ratio.

[0065] For example, step S10 may include: performing a first quantization inference on the neural network using a first quantization scheme, and determining a first outlier ratio of each operator according to a result of the first quantization inference; performing a second quantization inference on the neural network using a second quantization scheme, and determining a second outlier ratio of each operator according to a result of the second quantization inference.

[0066] For example, the first quantization scheme is different from the second quantization scheme, and the second quantization scheme stipulates that the second quantization inference is performed according to the original precision of the weight parameters and activation tensors of the neural network.

[0067] For example, the second quantization scheme may be the aforementioned W16A16 scheme, that is, both the weight parameters and the activation tensors use FP16 or BF16, that is, the second quantization inference is performed using the original precision of the weight parameters and the activation tensors.

[0068] For example, the first outlier ratio of an operator is the proportion of outliers in the activation tensor output by the operator during the first quantized inference process, and the second outlier ratio is the proportion of outliers in the activation tensor output by the operator during the second quantized inference process.

[0069] For example, in the first quantization scheme, at least one of the weight parameters and activation tensors of the neural network is quantized, for example, quantization inference is performed using a quantization accuracy lower than the original accuracy. For example, the first quantization scheme may be a quantization scheme with optimized performance relative to the second quantization scheme, for example, the performance is improved or the demand for video memory usage is optimized, but the first quantization scheme has a problem of precision loss.

[0070] For example, using a first quantization scheme to perform a first quantization inference on a neural network, and determining a first outlier ratio of each operator based on a result of the first quantization inference, can include: performing a first quantization inference according to a quantization scheme of weight parameters and activation tensors of the neural network specified by the first quantization scheme to obtain an output activation tensor of each operator, wherein multiple operators all use the same quantization scheme when performing the first quantization inference; for each operator, determining the number of outliers in the output activation tensor, and determining a ratio of the number to the total number of elements in the output activation tensor as the first outlier ratio.

[0071] For example, the first quantization scheme specifies the quantization scheme of each operator, such as how the weight parameters and activation tensors of each operator are quantized. Similarly, the second quantization scheme also specifies the quantization scheme of each operator, such as how the weight parameters and activation tensors of each operator are quantized.

[0072] For example, in the first quantization inference, all operators use the same quantization scheme, for example, the weight parameters of all operators are quantized to the same precision, and the activation tensors of all operators are quantized to the same precision.

[0073] For example, using a second quantization scheme to perform second quantization inference on the neural network and determining the second outlier ratio of each operator based on the result of the second quantization inference can include: performing second quantization inference according to the weight parameters of the neural network and the original precision of the activation tensor to obtain the output activation tensor of each operator; for each operator, determining the number of outliers in the output activation tensor, and determining the ratio of the number to the total number of elements in the output activation tensor as the second outlier ratio.

[0074] For example, in some other embodiments, the quantization reasoning may include only the second quantization reasoning, and the second outlier ratio obtained by the second quantization reasoning is used as the outlier ratio of the operator to adjust the first quantization scheme.

[0075] Through the second quantization inference, operators with more outliers can be identified. Since the output activation tensors of these operators have more outliers, the use of the low-bit first quantization scheme for quantization inference will result in a higher precision loss. Therefore, these operators are selected to adjust their quantization schemes, such as defining them as high-precision calculation operators and quantizing them using a high-precision quantization method to reduce the quantization error caused by quantization and significantly reduce the precision loss.

[0076] If the second quantization reasoning is used alone, in some cases, more operators in the neural network may be marked as high-precision calculation operators, which may make the performance regression of the reasoning operation more obvious. Therefore, in at least one embodiment of the present disclosure, the adjustment plan is determined in combination with the results of the first quantization reasoning and the second quantization reasoning. Specifically, if after the first quantization reasoning and the second quantization reasoning are executed, the proportion of outliers of certain operators in both quantization reasonings is high, then these operators are marked as high-precision calculation operators and quantized using a high-precision quantization method.

[0077] In some first quantization schemes, some technical means are used to reduce the precision loss, such as the aforementioned SmoothQuant method. Therefore, combined with the results of the first quantization reasoning, operators that really need to be defined as high-precision calculation operators can be obtained. These operators cannot achieve the expected quantization reasoning accuracy through the technical means of reducing precision loss provided by the first quantization scheme. Therefore, based on the results of the second quantization reasoning, the number of high-precision calculation operators can be further reduced in combination with the results of the first quantization reasoning, greatly reducing the precision loss while maintaining almost the same performance as the first quantization scheme as much as possible, and minimizing the performance regression of the reasoning process as much as possible.

[0078] For example, step S20 may include: classifying the operator into a first category of operators or a second category of operators based on a comparison result between the outlier ratio of the operator and a preset threshold, wherein the outlier ratio of the first category of operators is greater than or equal to the preset threshold; based on the first quantization scheme, adjusting the quantization scheme of the first category of operators, maintaining the quantization scheme of the second category of operators, and obtaining an adjusted first quantization scheme, wherein, in the adjusted first quantization scheme, the quantization accuracy of the activation tensor of the first category of operators is higher than or equal to the quantization accuracy of the activation tensor of the second category of operators.

[0079] For example, if the outlier ratio of an operator is greater than or equal to a preset threshold, the operator is classified as a first-class operator, for example, defined as a high-precision calculation operator; if the outlier ratio of an operator is less than the preset threshold, the operator is classified as a second-class operator, for example, defined as a low-precision calculation operator.

[0080] When adjusting the first quantization scheme, the quantization scheme of the first type of operator is adjusted, and the original quantization scheme of the second type of operator is maintained. Moreover, the quantization accuracy of the activation tensor of the adjusted first type of operator is higher than or equal to the quantization accuracy of the activation tensor of the second type of operator. In other words, during the adjustment process, at least the quantization accuracy of the activation tensor is improved to increase the range of values ​​that can be expressed, and reduce and avoid the loss of accuracy caused by ignoring outliers.

[0081] For example, if the quantitative reasoning includes the first quantitative reasoning and the second quantitative reasoning, and the outlier ratio includes the first outlier ratio and the second outlier ratio, classifying the operator as a first-category operator or a second-category operator based on the comparison result of the outlier ratio of the operator with a preset threshold value may include: in response to the first outlier ratio and the second outlier ratio of the operator being both higher than the preset threshold value, determining that the operator is a first-category operator; in response to at least one of the first outlier ratio and the second outlier ratio of the operator being lower than the preset threshold value, determining that the operator is a second-category operator.

[0082] For example, the setting of the preset threshold can be determined by statistical analysis of the large model. For example, through statistical analysis of the mainstream large model, it can be determined that once the proportion of outliers in the inference output tensor of an operator exceeds x%, x is an integer, this operator needs to be defined as a high-precision calculation operator, otherwise it will cause a large quantization error. Of course, according to different model structures, the preset threshold can also be adjusted accordingly, and the present disclosure does not make specific restrictions on this.

[0083] For example, in the adjusted first quantization scheme, the quantization scheme of the first type of operator stipulates that the activation tensor of the first type of operator is quantized to the original precision for inference operation. For example, the quantization precision of the activation tensor of the first type of operator is BF16 or FP16.

[0084] For example, in the adjusted first quantization scheme, the quantization scheme of the first type of operator stipulates that the weight parameter of the first type of operator is quantized to an 8-bit integer for inference operation. For example, the quantization precision of the weight parameter of the first type of operator is INT8.

[0085] For example, in response to the electronic device executing the data processing method supporting calculations in the FP8 data format, in the adjusted first quantization scheme, the quantization scheme for the first type of operator stipulates that weight parameters of the first type of operator are quantized to FP8 for inference operations.

[0086] According to practical experience, for large language models, the weight parameter representation range can achieve lossless accuracy using 8 bits, and less than 8 bits may cause irreversible loss of accuracy. Therefore, when adjusting the first quantization scheme, quantizing the weight parameters of the first type of operator to 8 bits can achieve lossless accuracy.

[0087] In one embodiment, if the first quantization scheme is the W8A8 INT8 SQ quantization scheme, for the first type of operator, the input and output activation tensors of the operator are defined as FP16 or BF16 calculations, and the weight parameters still maintain the quantization method of INT8 SmoothQuant. For the second type of operator, the input and output activation tensors and weight parameters of the operator both maintain the quantization method of INT8SmoothQuant.

[0088] In another embodiment, if the first quantization scheme is the W4A8 FP8 quantization scheme, for the first type of operator, the input and output activation tensors of the operator are defined as FP16 or BF16 calculations, and the weight parameters are quantized using FP8; for the second type of operator, the weight parameters are quantized using INT4, and the input and output activation tensors of the operator are quantized using FP8.

[0089] In another embodiment, if the first quantization scheme is the W4A16 AWQ quantization scheme, for the first type of operator, the input and output activation tensors of the operator are defined as FP16 or BF16 calculations, and the weight parameters are quantized using INT8; for the second type of operator, the weight parameters are quantized using INT4, and the input and output activation tensors of the operator are calculated using FP16 or BF16.

[0090] The same is true for other first quantization schemes, which are not listed here one by one.

[0091] After completing the above-mentioned automatic mixed precision quantization, the network graph of the entire network needs to be re-optimized due to the adjustment of the quantization scheme.

[0092] For example, at least one embodiment of the present disclosure further includes: optimizing a network graph of a neural network based on the adjusted first quantization scheme.

[0093] For example, in some embodiments, optimizing the network graph of a neural network based on the adjusted first quantization scheme may include: for multiple operators with contextual connections, performing sub-graph fusion on operators with consistent input and output precision according to the adjusted first quantization scheme.

[0094] This can reduce the performance loss and precision loss caused by numerical format conversion.

[0095] Figure 2A Schematic diagram of quantizing the precision map for a neural network.

[0096] For example, in a specific example, taking the Llama2-7b model as an example, the operator dependency diagram of the multi-layer perceptron (MLP) is as follows: Figure 2A As shown. Figure 2A As shown in the figure, the multilayer perceptron includes five operators, namely operator 1, operator 2, ..., operator 5. If the operators that perform linear calculations are quantized using the W8A8 INT8 SQ method and the operators that perform nonlinear calculations are quantized using the W16A16 method, then the five operators are all quantized using the W8A8 INT8 SQ method because they all perform linear calculations. This quantization reasoning method will result in a 7.6% loss in reasoning accuracy.

[0097] Figure 2B A schematic diagram of a neural network quantization precision mapping provided by an embodiment of the present disclosure.

[0098] Assuming that the first quantization scheme is W8A8 INT8, according to the data processing method provided by at least one embodiment of the present disclosure, the outlier ratio of each operator is counted, and the quantization scheme of the operator is determined according to the outlier ratio. The specific process is as described above and will not be repeated here. The schematic diagram of the adjusted first quantization scheme is shown in FIG. Figure 2B shown.

[0099] exist Figure 2B In the example, operator 1 and operator 2 are second-class operators, and still maintain W8A8 INT8 SQ quantization. Operator 3, operator 4, and operator 5 are first-class operators, and the activation tensor is calculated using F16 or BF16, and the weight parameter is quantized using INT8. After actual measurement, the adjusted first quantization scheme provided by the data processing method provided by at least one embodiment of the present disclosure makes the reasoning performance after quantization regress by 1%, but the overall accuracy loss is controlled at 0.2%, which greatly improves the accuracy of model quantization reasoning while maintaining low-bit quantization performance.

[0100] Afterwards, due to the adjustment of the operator quantization scheme, the entire network needs to be re-optimized. For example, Figure 2BIn the figure, operators 3, 4, and 5 have the same input and output precision, so they are fused into one subgraph. Operators 1 and 2 have no contextual relationship, so they are not fused into one subgraph.

[0101] The data processing method provided by at least one embodiment of the present disclosure can determine the quantization accuracy of the operator based on the outlier statistics of each operator output tensor, replacing manual experience in judging the operator quantization accuracy. For some extremely low-bit and high-performance quantization schemes such as W8A8 INT8 SQ, W4A8 FP8, W4A16 AWQ, etc., the accuracy of model quantization inference is improved while maintaining low-bit quantization performance.

[0102] Through actual measurement, the data processing method provided by at least one embodiment of the present disclosure uses the Llama2-7b large model as the neural network model, and the original evaluation accuracy in the MMLU (Massive Multitask Language Understanding) data set is 45.9%. After adopting the data processing method provided by at least one embodiment of the present disclosure, the performance regression and accuracy improvement of three low-bit quantization schemes, W8A8 INT8 SQ, W4A8 FP8, and W4A16 AWQ, are shown in the following Table 1:

[0103] Table 1

[0104]

[0105] As shown in the table above, the disclosed embodiments can significantly improve the reasoning accuracy and control the loss of reasoning accuracy within a range of almost lossless accuracy (the target accuracy loss can be controlled within 0.5%). In addition, the performance regression caused by maintaining high-precision calculations can also be controlled within 5%.

[0106] At least one embodiment of the present disclosure further provides a data processing device, Figure 3 A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure.

[0107] like Figure 3 As shown, the data processing device 200 may include a quantization reasoning training module 201, an adjustment module 202 and a reasoning module 203. It should be noted that Figure 3 The components and structure of the data processing device 200 shown are merely exemplary and non-limiting. The data processing device 200 may also have other components and structures as required.

[0108] For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination of the two, etc. The following embodiments are the same and will not be described in detail. For example, these units can be implemented by a central processing unit (CPU), a data processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions.

[0109] For example, the data processing device may be a processor that needs to perform reasoning operations, such as a general-purpose processor such as a central processing unit, a multi-core processor, a graphics processor, a general-purpose graphics processor, a digital signal processor, or a special-purpose processor such as an AI processor, a tensor processing unit (TPU), and a field programmable gate array (FPGA). For example, the data processing device may be a component specially provided in some GPUs and AI accelerators for optimizing calculations, such as a deep learning accelerator. Of course, the present disclosure is not limited to this, and the data processing device may be any device that needs to execute a data processing method to perform corresponding reasoning operations.

[0110] For example, the quantitative reasoning training module 201 is configured to perform quantitative reasoning on the neural network and determine the outlier ratio of each operator among the multiple operators included in the neural network, wherein the outlier ratio of the operator is the proportion of outliers in the output activation tensor output by the operator during the quantitative reasoning process.

[0111] For example, the adjustment module 202 is configured to adjust a first quantization scheme according to the outlier ratio of each operator, wherein in the first quantization scheme, the quantization accuracy of at least one of the weight parameters and activation tensors of the neural network is lower than the original accuracy.

[0112] For example, the inference module 203 is configured to perform inference operations of the neural network according to the adjusted first quantization scheme.

[0113] For example, the multiple operators include a first operator and a second operator, and the adjusted first quantization scheme stipulates that when performing an inference operation, the quantization precision of at least one of the activation tensor and the weight parameter of the first operator and the second operator is different.

[0114] For example, the outlier ratio of the first operator is greater than or equal to a preset threshold, the outlier ratio of the second operator is less than the preset threshold, and the quantization accuracy of the activation tensor of the first operator is higher than or equal to the quantization accuracy of the activation tensor of the second operator. For example, the quantization accuracy of the activation tensor of the first operator is the original accuracy, and the quantization accuracy of the activation tensor of the second operator is lower than or equal to the original accuracy.

[0115] For example, the quantized reasoning training module 201, the adjustment module 202, and the reasoning module 203 may include codes and programs stored in a memory; the processor may execute the codes and programs to implement some or all of the functions of the quantized reasoning training module 201, the adjustment module 202, and the reasoning module 203 as described above. For example, the quantized reasoning training module 201, the adjustment module 202, and the reasoning module 203 may be dedicated hardware devices used to implement some or all of the functions of the quantized reasoning training module 201, the adjustment module 202, and the reasoning module 203 as described above. For example, the quantized reasoning training module 201, the adjustment module 202, and the reasoning module 203 may be a circuit board or a combination of multiple circuit boards for implementing the functions described above. In an embodiment of the present application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) firmware stored in the memory that is executable by the processor.

[0116] It should be noted that the quantized reasoning training module 201 can be used to implement Figure 1 In step S10, the adjustment module 202 can be used to implement Figure 1 In step S20 shown in FIG. 1 , the reasoning module 203 can be used to implement Figure 1 Step S30 shown. Therefore, for the specific description of the functions that can be realized by the quantitative reasoning training module 201, reference can be made to the relevant description of step S10 in the embodiment of the above-mentioned data processing method. For the specific description of the functions that can be realized by the adjustment module 202, reference can be made to the relevant description of step S20 in the embodiment of the above-mentioned data processing method. For the specific description of the functions that can be realized by the reasoning module 203, reference can be made to the relevant description of step S30 in the embodiment of the above-mentioned data processing method. The repeated parts will not be repeated.

[0117] The data processing method provided by at least one embodiment of the present disclosure can determine the quantization accuracy of the operator by using the outlier statistics of each operator output tensor, instead of judging the quantization accuracy of the operator based on manual experience. For some extremely low-bit and high-performance quantization schemes such as W8A8 INT8 SQ, W4A8 FP8, W4A16 AWQ, etc., the accuracy of model quantization inference is improved while maintaining low-bit quantization performance.

[0118] The data processing device 200 can achieve technical effects similar to those of the aforementioned data processing method, which will not be described in detail here.

[0119] It should be noted that in the embodiments of the present disclosure, the data processing device 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or in other applicable ways.

[0120] Figure 4 The following is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 4 As shown, the electronic device 300 is suitable for implementing the data processing method provided by the embodiment of the present disclosure. It should be noted that Figure 4 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0121] For example, Figure 4 As shown, in some examples, the electronic device 300 includes a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to achieve various functions, such as executing the data processing method provided by at least one embodiment of the present disclosure.

[0122] For example, when the computer readable instructions are executed by the processing device 301, one or more steps in the data processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the data processing method, reference may be made to the relevant description in the above data processing method embodiment, and the repetitive parts will not be repeated.

[0123] Various programs and data required for the operation of the computer system are also stored in the RAM 303 . The processing device 301 , the ROM 302 , and the RAM 303 are connected to each other via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0124] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory (cache), etc. For example, computer-readable instructions may be loaded from storage device 308 into random access memory (RAM) 303 to run computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by applications, etc., may also be stored in the computer-readable storage medium.

[0125] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device 300 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown, and the electronic device 300 may be implemented or have more or fewer devices instead. For example, the processor 301 can control other components in the electronic device 300 to perform the desired functions. The processing device 301 can be a device with data processing capabilities and / or program execution capabilities such as a central processing unit (CPU), a tensor processing unit (TPU) or a graphics processing unit (GPU). The central processing unit can be an X86 or ARM architecture, etc. The graphics processor can be directly integrated into the motherboard separately, or built into the north bridge chip of the motherboard. The graphics processor can also be built into the central processing unit.

[0126] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 309 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet and / or a wireless network such as a cellular phone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN). Wireless communication may use any of a variety of communication standards, protocols, and techniques, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0127] For example, the electronic device may be any device such as a mobile phone, a tablet computer, a laptop computer, an e-book, a game console, a television, a digital photo frame, a navigator, or any combination of electronic devices and hardware, which is not limited in the embodiments of the present disclosure.

[0128] For example, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned data processing method defined in the method of the embodiment of the present disclosure is executed.

[0129] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0130] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0131] Figure 5 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 5 As shown, the storage medium 400 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 401 may be non-transitory stored on the storage medium 400. For example, when the computer-readable instructions 401 are executed by a processor, the data processing method described above may be executed.

[0132] For example, the storage medium 400 may be applied to the above-mentioned electronic device. For example, the storage medium 400 may include a memory in the electronic device.

[0133] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0134] For example, the description of the storage medium 400 may refer to the description of the memory in the embodiment of the electronic device, and the repeated parts will not be repeated.

[0135] Figure 6 A schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure. The electronic device provided by the present disclosure can be applied in an Internet system.

[0136] use Figure 6 The computer system provided in can implement the functions of the data processing device and / or electronic device involved in the present disclosure. Such computer systems may include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smart watches, smart rings, smart helmets and any smart portable devices or wearable devices. The specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. This computer device can be a general-purpose computer device or a computer device with a specific purpose. Both computer devices can be used to implement the data processing device and / or electronic device in this embodiment. The computer system may include any components that implement the information required to implement the data processing currently described. For example, the computer system can be implemented by a computer device through its hardware devices, software programs, firmware and their combinations. For convenience, Figure 6 Only one computer device is drawn in the figure, but the relevant computer functions of the information required for realizing data processing described in this embodiment can be implemented in a distributed manner by a group of similar platforms to disperse the processing load of the computer system.

[0137] like Figure 6As shown, the computer system may include a communication port 250, connected to which is a network for realizing data communication. For example, the computer system can send and receive information and data through the communication port 250, that is, the communication port 250 can realize wireless or wired communication between the computer system and other electronic devices to exchange data. The computer system may also include a processor group 220 (i.e., the processing device described above) for executing program instructions. The processor group 220 may be composed of at least one processor (e.g., a CPU or a GPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage medium described above), such as a hard disk 270, a read-only memory (ROM) 230, and a random access memory (RAM) 240, which can be used to store various data files used for computer processing and / or communication, and possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260, which is used to realize input / output data flow between the computer system and other components (e.g., a user interface 280, etc.).

[0138] Typically, the following devices can be connected to the input / output component 260: input devices such as a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices such as a display (e.g., an LCD, an OLED display, etc.), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and a communication interface.

[0139] Although Figure 6 A computer system having various devices is shown, but it should be understood that the computer system is not required to have all of the devices shown, and the computer system may alternatively have more or fewer devices.

[0140] Those skilled in the art will appreciate that the contents disclosed in this disclosure may be subject to various modifications and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0141] In addition, although the present disclosure makes various references to certain units in the system according to embodiments of the present disclosure, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0142] Flowcharts are used in this disclosure to illustrate the steps of the method according to the embodiments of the present disclosure. It should be understood that the preceding or following steps are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes.

[0143] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of a software functional module. The present disclosure is not limited to any particular form of combination of hardware and software.

[0144] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or highly formal sense, unless explicitly defined as such herein.

[0145] The above is an explanation of the present disclosure and should not be considered as a limitation thereof. Although several exemplary embodiments of the present disclosure are described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims. It should be understood that the above is an explanation of the present disclosure and should not be considered to be limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. A data processing method, comprising: Performing quantitative reasoning on a neural network, determining the outlier ratio of each operator among a plurality of operators included in the neural network, wherein the outlier ratio of the operator is the proportion of outliers in an output activation tensor output by the operator during the quantitative reasoning process, the neural network includes a large language model, and the data used in the quantitative reasoning includes data related to language understanding and used for natural language processing; Adjusting a first quantization scheme according to the outlier ratio of each operator, wherein, in the first quantization scheme, a quantization accuracy of at least one of a weight parameter and an activation tensor of the neural network is lower than an original accuracy; Performing an inference operation of the neural network according to the adjusted first quantization scheme; The first quantization scheme is adjusted according to the outlier ratio of each operator, including: According to the comparison result of the outlier ratio of the operator and the preset threshold, the operator is classified into a first type of operator or a second type of operator, wherein the outlier ratio of the first type of operator is greater than or equal to the preset threshold; Based on the first quantization scheme, adjusting the quantization scheme of the first type of operators, maintaining the quantization scheme of the second type of operators, and obtaining the adjusted first quantization scheme, Among them, in the adjusted first quantization scheme, the quantization accuracy of the activation tensor of the first type of operator is higher than or equal to the quantization accuracy of the activation tensor of the second type of operator.

2. The data processing method according to claim 1, wherein: The adjusted first quantization scheme stipulates that when performing the inference operation, the quantization precision of the activation tensor of the first type of operator is different from the quantization precision of the activation tensor of the second type of operator, and / or the quantization precision of the weight parameter of the first type of operator is different from the weight parameter of the second type of operator.

3. The data processing method according to claim 2, wherein: The outlier ratio of the second type of operators is less than the preset threshold, The quantization accuracy of the activation tensor of the first type of operator is the original accuracy, and the quantization accuracy of the activation tensor of the second type of operator is lower than or equal to the original accuracy.

4. The data processing method according to claim 1, wherein: The quantitative reasoning includes a first quantitative reasoning and a second quantitative reasoning, The outlier ratio includes a first outlier ratio and a second outlier ratio. Performing quantitative reasoning on the neural network to determine the outlier ratio of each operator in a plurality of operators included in the neural network includes: Performing the first quantitative reasoning on the neural network using the first quantization scheme, and determining the first outlier ratio of each operator according to a result of the first quantitative reasoning; Performing the second quantization reasoning on the neural network using a second quantization scheme, and determining the second outlier ratio of each operator according to a result of the second quantization reasoning; Among them, the first quantization scheme is different from the second quantization scheme, and the second quantization scheme stipulates that the second quantization inference is performed according to the original precision of the weight parameters of the neural network and the activation tensor of the neural network.

5. The data processing method according to claim 4, wherein: The first quantization scheme is used to perform the first quantization reasoning on the neural network, and the first outlier ratio of each operator is determined according to the result of the first quantization reasoning, including: Performing the first quantization inference according to the quantization scheme of the weight parameters of the neural network and the activation tensor of the neural network specified by the first quantization scheme to obtain the output activation tensor of each operator, wherein the multiple operators all adopt the same quantization scheme when performing the first quantization inference; For each operator, determine the number of outliers in the output activation tensor, and determine a ratio of the number to the total number of elements in the output activation tensor as the first outlier ratio.

6. The data processing method according to claim 4, wherein: The second quantization scheme is used to perform second quantization reasoning of the neural network, and a second outlier ratio of each operator is determined according to a result of the second quantization reasoning, including: Performing the second quantized inference according to the weight parameters of the neural network and the original precision of the activation tensor of the neural network to obtain an output activation tensor of each operator; For each operator, the number of outliers in the output activation tensor is determined, and a ratio of the number to the total number of elements in the output activation tensor is determined as the second outlier ratio.

7. The data processing method according to claim 1, wherein: In response to the quantitative reasoning including the first quantitative reasoning and the second quantitative reasoning, the outlier ratio including the first outlier ratio and the second outlier ratio, Classifying the operator into a first type of operator or a second type of operator according to a comparison result between the outlier ratio of the operator and a preset threshold value includes: In response to the first outlier ratio and the second outlier ratio of the operator being both higher than the preset threshold, determining that the operator is the first type of operator; In response to at least one of the first outlier ratio and the second outlier ratio of the operator being lower than the preset threshold, the operator is determined to be an operator of the second type.

8. The data processing method according to claim 1, wherein: In the adjusted first quantization scheme, the quantization scheme of the first type of operators stipulates that the activation tensors of the first type of operators are quantized to the original precision for the inference operation.

9. The data processing method according to claim 1, wherein: In the adjusted first quantization scheme, the quantization scheme of the first type of operators stipulates that the weight parameters of the first type of operators are quantized into 8-bit integers for performing the inference operation.

10. The data processing method according to claim 1, wherein: In response to the electronic device executing the data processing method supporting calculations in the FP8 data format, in the adjusted first quantization scheme, the quantization scheme of the first type of operators stipulates that weight parameters of the first type of operators are quantized to FP8 for performing the inference operation.

11. The data processing method according to any one of claims 1 to 10, further comprising: Based on the adjusted first quantization scheme, a network graph of the neural network is optimized.

12. The data processing method according to claim 11, wherein: Based on the adjusted first quantization scheme, optimizing the network graph of the neural network includes: For multiple operators with contextual connections, operators with consistent input and output precision are sub-graph fused according to the adjusted first quantization scheme.

13. An electronic device, comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the data processing method according to any one of claims 1-12 is implemented.

14. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 12 is implemented.

Images