Method, apparatus, and related product for processing data

Abstract

Embodiments of the present disclosure relate to a method and apparatus for processing data and related products. Embodiments of the present disclosure relate to a board card comprising a memory device, an interface device, a control device, and an artificial intelligence chip; wherein the artificial intelligence chip is connected with the memory device, the control device, and the interface device respectively; the memory device is configured to store data; the interface device is configured to realize data transmission between the artificial intelligence chip and an external device; and the control device is configured to monitor a state of the artificial intelligence chip. The board card can be used to perform artificial intelligence operation.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to the field of computer technology, and more particularly to a method, an apparatus and a related product for processing data. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, its application field is more and more extensive, and it has been well applied in the fields of image recognition, speech recognition, natural language processing and the like. However, with the increase of complexity and accuracy of artificial intelligence algorithm, the machine learning model is getting larger and larger, and the amount of data to be processed is also getting larger and larger. When a large amount of data processing is performed, a large amount of computation and time is required, and the processing efficiency is low. SUMMARY

[0003] In view of this, embodiments of the present disclosure provide a method, an apparatus and a related product for processing data.

[0004] In a first aspect of the present disclosure, a method for processing data is provided. The method comprises: obtaining a set of to-be-quantized data for a machine learning model; determining a plurality of sets of quantized data by respectively quantizing the set of to-be-quantized data using a plurality of pairs of truncation thresholds, wherein each pair of truncation thresholds in the plurality of pairs of truncation thresholds comprises a symmetric positive truncation value and a negative truncation value; and selecting a pair of truncation thresholds from the plurality of pairs of truncation thresholds for quantizing the set of to-be-quantized data based on a difference between a mean value of absolute values of each set of quantized data in the plurality of sets of quantized data and a mean value of absolute values of the set of to-be-quantized data.

[0005] In a second aspect of the present disclosure, an apparatus for processing data is provided. The apparatus comprises: a to-be-quantized data obtaining unit configured to obtain a set of to-be-quantized data for a machine learning model; a quantized data determining unit configured to determine a plurality of sets of quantized data by respectively quantizing the set of to-be-quantized data using a plurality of pairs of truncation thresholds, wherein each pair of truncation thresholds in the plurality of pairs of truncation thresholds comprises a symmetric positive truncation value and a negative truncation value; and a truncation threshold selecting unit configured to select a pair of truncation thresholds from the plurality of pairs of truncation thresholds for quantizing the set of to-be-quantized data based on a difference between a mean value of absolute values of each set of quantized data in the plurality of sets of quantized data and a mean value of absolute values of the set of to-be-quantized data.

[0006] In a third aspect of the present disclosure, a computer readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to carry out the method according to the various embodiments of the present disclosure.

[0007] In a fourth aspect of the present disclosure, an artificial intelligence chip comprising the apparatus for processing data according to the various embodiments of the present disclosure is provided.

[0008] In a fifth aspect of the present disclosure, an electronic device is provided, which includes an artificial intelligence chip according to various embodiments of the present disclosure.

[0009] In a sixth aspect of the present disclosure, a board card is provided, which includes a storage device, an interface device, a control device, and an artificial intelligence chip according to various embodiments of the present disclosure. The artificial intelligence chip is connected to the storage device, the control device, and the interface device. The storage device is configured to store data. The interface device is configured to implement data transmission between the artificial intelligence chip and an external device. The control device is configured to monitor a state of the artificial intelligence chip.

[0010] The technical features in the claims can deduce the beneficial effects on the technical problems in the background art. Other features and aspects of the present disclosure will become apparent from the detailed description of exemplary embodiments with reference to the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0012] Figure 1 A schematic diagram of a processing system for processing data according to an embodiment of the present disclosure is shown;

[0013] Figure 2 A schematic diagram of an example architecture of a neural network according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram of a process for quantizing data according to an embodiment of the present disclosure is shown;

[0015] Figure 4A A schematic diagram of a process for symmetrically quantizing data according to an embodiment of the present disclosure is shown;

[0016] Figure 4B A schematic diagram of a process for symmetrically quantizing data based on a truncation threshold according to an embodiment of the present disclosure is shown;

[0017] Figure 5 A flowchart of a method for processing data according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A flowchart of a method for searching a truncation threshold for symmetric quantization according to an embodiment of the present disclosure is shown;

[0019] Figure 7A A schematic diagram of a process for coarsely searching a truncation threshold for symmetric quantization according to an embodiment of the present disclosure is shown;

[0020] Figure 7B A diagram showing a method for searching a truncation threshold for fine-grained search symmetric quantization according to embodiments of the present disclosure is shown;

[0021] Figure 8 A flowchart showing a method for iteratively searching for an optimal truncation threshold according to embodiments of the present disclosure is shown;

[0022] Figure 9 A block diagram showing an apparatus for processing data according to embodiments of the present disclosure is shown; and

[0023] Figure 10 A structural block diagram of a board card according to embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0025] It should be understood that the terms “first”, “second”, “third”, and “fourth” in the claims, specification, and drawings of the present disclosure are used to distinguish different objects, rather than to describe a particular order. The terms “include” and “contain” used in the specification and claims of the present disclosure indicate the presence of the described features, whole, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components, and / or sets thereof.

[0026] It should also be understood that the terms used in the present disclosure specification are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used in the present disclosure specification and claims, unless otherwise clear from the context, the singular forms “a”, “an”, and “the” are intended to include plural forms. It should be further understood that the term “and / or” used in the present disclosure specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0027] As used in the specification and claims, the term “if’ can be interpreted as meaning “when,” or “upon,” or “in response to a determination,” or “in response to a detection” depending on the context. Similarly, the phrase “if it is determined” or “if [the recited condition or event] is detected” can be interpreted as meaning “upon determining” or “in response to a determining” or “upon detecting [the recited condition or event]” or “in response to a detection of [the recited condition or event],” depending on the context.

[0028] Generally, when quantizing data, if the selected range of values is wide, the precision of the quantized data is low, and if the range of values is too small, too much data is truncated, resulting in loss of information of the data distributed on both sides. Therefore, it is necessary to find a pair of appropriate truncation thresholds to quantize the data so that the loss of data quantization is minimal or small. Traditionally, the best truncation threshold is determined by the KL divergence (Kullback-Leibler divergence) method, which can determine the correlation between the data before and after quantization. The KL divergence is also known as relative entropy, information divergence, and information gain. The KL divergence is a measure of the difference between two probability distributions P and Q. Assuming that the distribution of 32-bit floating-point numbers before quantization is P and the distribution of 8-bit integers after quantization is Q, the smaller the KL divergence between P and Q, the closer the distribution before and after quantization, and the more effective the quantization. However, the inventors of the present application found that the quantization effect achieved by the traditional KL method is not good, and usually causes a large loss of precision.

[0029] To this end, embodiments of the present disclosure propose a new scheme for determining truncation thresholds for symmetric quantization, which can achieve smaller loss of quantization precision than traditional techniques (such as the KL method). According to embodiments of the present disclosure, after obtaining a set of data to be quantized for a machine learning model, a plurality of sets of quantized data are determined by quantizing the set of data to be quantized using a plurality of pairs of truncation thresholds, wherein each pair of truncation thresholds in the plurality of pairs of truncation thresholds includes a symmetric positive truncation threshold and a symmetric negative truncation threshold. Then, a suitable pair of truncation thresholds is selected from the plurality of pairs of truncation thresholds using the difference between the mean of the absolute values of each set of quantized data and the mean of the absolute values of the set of data to be quantized as an evaluation index. In this way, a more suitable pair of truncation thresholds can be found.

[0030] The following references are incorporated by reference in their entirety: Figures 1 to 10The following detailed description illustrates the basic principles and several example implementations of the present disclosure. It should be understood that the example embodiments are given only for the purpose of better illustrating the embodiments of the present disclosure and not to limit the scope of the present disclosure in any way.

[0031] Figure 1 A schematic diagram of a processing system 100 for processing data according to an embodiment of the present disclosure is shown. As shown, the processing system 100 includes a plurality of processors 101-1, 101-2, 101-3 (collectively referred to as processors 101) for executing sequences of instructions and a memory 102 for storing data, which can include a random access memory (RAM) and a register file. The plurality of processors 101 in the processing system 100 can share part of the memory space, such as a shared part of the RAM memory space and the register file, and can also have their own memory space at the same time. Figure 1

[0032] It should be understood that the various methods according to embodiments of the present disclosure can be applied to any one of the plurality of processors (multi-core) of the processing system 100 (such as an artificial intelligence chip). The processor can be a general-purpose processor, such as a CPU (Central Processing Unit), or an artificial intelligence processor (IPU) for performing artificial intelligence operations. The artificial intelligence operations can include machine learning operations, brain-like operations, etc. The machine learning operations include neural network operations, k-means operations, support vector machine operations, etc. The artificial intelligence processor can include one or a combination of a GPU (Graphics Processing Unit), a NPU (Neural-Network Processing Unit), a DSP (Digital Signal Process), and an FPGA (Field-Programmable Gate Array) chip, for example. The present disclosure does not limit the specific type of the processor. In addition, the types of the plurality of processors in the processing system 100 can be the same or different, and the present disclosure does not limit this.

[0033] In one possible implementation, the processor mentioned in the present disclosure can include a plurality of processing units, each of which can independently run various tasks assigned to it, such as convolution operation tasks, pooling tasks, or fully connected tasks, etc. The present disclosure does not limit the processing units and the tasks run by the processing units.

[0034] Figure 2 ​A schematic diagram of an example architecture of a neural network 200 according to an embodiment of the present disclosure is shown. A neural network (NN) is a mathematical model that mimics the structure and function of biological neural networks. A neural network performs computations through a large number of interconnected neurons. Therefore, a neural network is a computational model consisting of a large number of interconnected nodes (or "neurons"). Each node represents a specific output function called an activation function. Each connection between two neurons represents a weighted value of the signal passing through that connection, called a weight, which is equivalent to the memory of the neural network. The output of the neural network varies depending on the connections between neurons and the weights and activation functions. In a neural network, a neuron is the basic unit. It receives a certain number of inputs and a bias, which is multiplied by a weight when a signal (value) arrives. A connection connects a neuron to another neuron in another layer or within the same layer, and the connection is accompanied by an associated weight. Additionally, the bias is an extra input to the neuron; it is always 1 and has its own connection weight. This ensures that the neuron is activated even if all inputs are empty (all 0s).

[0035] In applications, if a non-linear function isn't applied to the neurons in a neural network, the network is merely a linear function and no more powerful than a single neuron. If we want the output of a neural network to be between 0 and 1—for example, in cat / dog identification—outputs close to 0 can be interpreted as cats, and outputs close to 1 as dogs. To achieve this, activation functions, such as the sigmoid activation function, are introduced into the neural network. Regarding this activation function, all we need to know is that its return value is a number between 0 and 1. Therefore, activation functions introduce non-linearity into the neural network, narrowing the results of the network's computation to a smaller range. In reality, how the activation function is expressed is not important; what matters is parameterizing a non-linear function with weights, which can be changed to alter the non-linear function.

[0036] like Figure 2 The diagram shown is a schematic representation of the structure of neural network 200. Figure 2 The neural network shown includes three layers: an input layer 210, a hidden layer 220, and an output layer 230. Figure 2 The hidden layer 220 shown has three layers; however, it can also include more or fewer layers. The neurons in the input layer 210 are called input neurons. As the first layer in the neural network, the input layer requires input signals (values) and passes them to the next layer. It does not perform any operations on the input signals (values) and has no associated weights or biases. Figure 2In the illustrated neural network, 4 input signals (values) can be received.

[0037] The hidden layers 220 are used to apply different transformations to the input data by neurons (nodes). A hidden layer is a collection of neurons (Representation) arranged vertically. In the illustrated neural network, the first hidden layer has 4 neurons (nodes), the second layer has 6 neurons, and the third layer has 3 neurons. Finally, the hidden layers pass values to the output layer. Figure 2 In the illustrated neural network, there are 3 hidden layers. The first hidden layer has 4 neurons (nodes), the second layer has 6 neurons, and the third layer has 3 neurons. Finally, the hidden layers pass values to the output layer. Figure 2 The illustrated neural network 200 has full connections between each neuron in the 3 hidden layers, i.e., each neuron in the 3 hidden layers is connected to every neuron in the next layer. It should be noted that not every neural network has fully connected hidden layers.

[0038] The neurons of the output layer 230 are called output neurons. The output layer receives the output from the last hidden layer. Through the output layer 230, the desired value and the desired range can be determined. In the illustrated neural network, the output layer has 3 neurons, i.e., there are 3 output signals (values). Figure 2 In the illustrated neural network, the output layer has 3 neurons, i.e., there are 3 output signals (values).

[0039] In practical applications, the role of a neural network is to be trained in advance with a large amount of sample data (containing input and output), and after training, the neural network is used to obtain an accurate output for future real environment inputs.

[0040] Before starting to discuss the training of a neural network, a loss function needs to be defined. The loss function is a function that indicates how well a neural network performs on a certain task. The most direct way to do this is to, during the training process, pass a number along the neural network for each sample data, and then square the difference between this number and the actual number that is desired to be obtained. This calculated distance between the predicted value and the true value is the distance or loss function value that the neural network is trained to reduce.

[0041] When starting to train a neural network, the weights are randomly initialized. Obviously, the initialized neural network will not provide a good result. During the training process, it is assumed that a very poor neural network is started with, and through training, a network with high accuracy can be obtained. At the same time, it is also desired that at the end of the training, the function value of the loss function becomes very small.

[0042] The training process of the neural network is divided into two stages. The first stage is the forward processing of the signal, from the input layer 210 through the hidden layer 220, and finally to the output layer 230. The second stage is the backward propagation of the gradient, from the output layer 230 to the hidden layer 220, and finally to the input layer 210, according to the gradient to adjust the weight and bias of each layer in the neural network in turn.

[0043] In the process of forward processing, the input value is input to the input layer 210 of the neural network, and the output called the predicted value is obtained from the output layer 230 of the neural network. When the input value is provided to the input layer 210 of the neural network, it does not perform any operation. In the hidden layer, the second hidden layer obtains the predicted intermediate result value from the first hidden layer and performs the calculation operation and the activation operation, and then passes the obtained predicted intermediate result value to the next hidden layer. The same operation is performed in the subsequent layers, and finally the output value is obtained at the output layer 230 of the neural network.

[0044] After the forward processing, an output value called the predicted value is obtained. In order to calculate the error, a loss function is used to compare the predicted value with the actual output value to obtain the corresponding error value. Backpropagation uses the chain rule of differential calculus, in which the derivative of the error value corresponding to the last layer weight of the neural network is first calculated. These derivatives are called gradients, and then these gradients are used to calculate the gradients of the second-to-last layer in the neural network. This process is repeated until the gradients of each weight in the neural network are obtained. Finally, the corresponding gradients are subtracted from the weights, thereby updating the weights once to reduce the error value.

[0045] In addition, for the neural network, fine-tuning is to load the trained neural network, and the fine-tuning process is the same as the training process, which is divided into two stages. The first stage is the forward processing of the signal, and the second stage is the backward propagation of the gradient to update the weight of the trained neural network. The difference between training and fine-tuning is that training is to randomly process the initialized neural network, and training the neural network from scratch, while fine-tuning is not training from scratch.

[0046] In the process of training or fine-tuning of the neural network, the weight value in the neural network is updated by gradient once per forward processing of the signal and corresponding backward propagation of the error, which is called an iteration. In order to obtain a neural network with expected accuracy, a very large sample data set is required in the training process. In this case, it is impossible to input the sample data set into the computer at one time. Therefore, in order to solve this problem, the sample data set needs to be divided into multiple blocks, each block is transmitted to the computer, and the weight value of the neural network is updated once after each block of data set is forward processed. When a complete sample data set is forward processed once in the neural network and the corresponding weight value is returned once, this process is called an epoch. In practice, it is not enough to transmit a complete data set in the neural network, and the complete data set needs to be transmitted multiple times in the same neural network, that is, multiple epochs are required to finally obtain a neural network with expected accuracy.

[0047] In the process of training or fine-tuning of the neural network, it is generally desired to be faster and more accurate. The data of the neural network is represented by high-precision data format, such as floating-point numbers, so in the training or fine-tuning process, the data involved are all high-precision data format, and then the trained neural network is quantized. Taking the weight value of the entire neural network as the quantization object and the quantized weight value as an 8-bit fixed-point number as an example, since there are usually millions of connections in a neural network, almost all the space is occupied by the weight values of the neuron connections. Moreover, these weight values are different floating-point numbers. The weight values of each layer tend to be normally distributed in a certain interval, for example, (-3.0, 3.0). The maximum and minimum values of the weight values of each layer in the neural network are saved, and each floating-point value is represented by an 8-bit fixed-point number. Among them, 256 quantization intervals are linearly divided in the range of the maximum and minimum values, and each quantization interval is represented by an 8-bit fixed-point number. For example: in the interval (-3.0, 3.0), byte 0 represents -3.0, and byte 255 represents 3.0. In this way, byte 128 represents 0.

[0048] For data represented by high-precision data format, taking floating-point numbers as an example, according to the computer architecture, the operation representation rules based on floating-point numbers and fixed-point numbers, for fixed-point operations and floating-point operations of the same length, the floating-point operation mode is more complex, and more logic devices are required to form a floating-point operation unit. In this way, the volume of the floating-point operation unit is larger than that of the fixed-point operation unit. Moreover, the floating-point operation unit consumes more resources for processing, so that the power consumption difference between fixed-point operation and floating-point operation is usually of orders of magnitude. In short, the chip area and power consumption occupied by the floating-point operation unit are much larger than those of the fixed-point operation unit.

[0049] Figure 3 A schematic diagram of a process 300 for quantizing data according to an embodiment of the present disclosure is shown. Referring to FIG. 3, Figure 3 , the input data 310 is an unquantized floating point number, for example, a 32-bit floating point number. If the input data 310 is directly input into a neural network model 340 for processing, it will consume more computing resources and the processing speed will be slower. Therefore, at block 320, the input data can be quantized to obtain quantized data 330 (for example, 8-bit integers). If the quantized data 330 is input into the neural network model 340 for processing, since 8-bit integer calculation is faster, the neural network model 340 will complete the processing of the input data more quickly and generate a corresponding output result 350.

[0050] In the quantization process from the unquantized input data 310 to the quantized data 330, some accuracy loss will be caused to some extent, and the degree of accuracy loss will directly affect the accuracy of the output result 350. Therefore, in the quantization process of the input data 330, it is necessary to ensure that the accuracy loss of the quantization process is minimal or as small as possible.

[0051] Figure 4A A diagram 400 for symmetrically quantizing data according to an embodiment of the present disclosure is shown. As Figure 4A shown, it is the simplest symmetric quantization method, which directly selects the absolute value maximum value of all values in the data to be quantized, that is, |max|, and then quantizes in the range of -|max| to |max| to generate quantized data. However, this method does not make any truncation, which will result in lower accuracy of the quantized data.

[0052] Figure 4B A diagram 450 for symmetrically quantizing data based on a truncation threshold according to an embodiment of the present disclosure is shown. Unlike Figure 4A the direct quantization method in Figure 4B , a truncation threshold T is selected in Figure 4B . Data outside the range of -|T| to |T| will be set to -|T| or |T|. For example, in the example of , the 3 values to be quantized in the circle 460 will be quantized as the value -|T| because they are outside the truncation range, and quantized into the data point 470. In this way, by using the truncation threshold to reduce the value range of the data to be quantized, the accuracy of the quantized data can be improved. However, how to obtain a truncation threshold with minimal quantization accuracy loss is a technical problem to be solved.

[0053] Figure 5 A flowchart of a method 500 for processing data according to an embodiment of the present disclosure is shown. It should be understood that the method 500 can be performed by the neural network model 340 of FIG. 3, Figure 1The one or more processors 101 described are configured to perform.

[0054] At block 502, a set of data to be quantized for a machine learning model is obtained. For example, with reference to the above description of FIG. 1, the input data 310 to be quantized can be obtained, and the input data is quantized to speed up the processing of the neural network model 340. In addition, some parameters of the neural network model itself, such as weights, can also be quantized, and by quantizing the network parameters, the size of the neural network model can be reduced. In some embodiments, the data to be quantized can be 32-bit floating-point numbers. Alternatively, the data to be quantized can also be floating-point numbers of other bit numbers, or other data types. Figure 3

[0055] In the symmetric quantization scheme, the clipping threshold is a pair of symmetric positive and negative values, i.e., the clipping positive value and the clipping negative value, which are the same in value but opposite in sign.

[0056] According to embodiments of the present disclosure, a plurality of pairs of clipping thresholds can be selected to quantize the data to be quantized. In some embodiments, some clipping thresholds can be selected at fixed intervals, for example, according to the maximum absolute value in the data to be quantized, a clipping threshold is selected every predetermined distance. In some embodiments, only a few clipping thresholds at specific positions can be selected, for example, only a few predetermined proportion of the values of the maximum absolute value.

[0057] In some embodiments, one or more quantization parameters can be calculated according to each pair of clipping thresholds, and then the calculated quantization parameters are used to quantize the data to be quantized. Alternatively, the data to be quantized can also be quantized directly according to the clipping threshold by various formulas or models without separately calculating the values of the quantization parameters.

[0058] At block 506, based on the difference between the mean of the absolute values of each set of quantized data and the mean of the absolute values of the set of data to be quantized, a pair of clipping thresholds is selected from the plurality of pairs of clipping thresholds for quantizing the set of data to be quantized. The inventors of the present application have found through research and a large number of experiments that the difference in the mean of the absolute values of the data before and after quantization can reflect the accuracy loss before and after quantization, and the smaller the difference in the mean of the absolute values, the smaller the accuracy loss of the quantization operation. Therefore, the embodiments of the present disclosure use the difference in the mean of the absolute values of the data before and after quantization as an indicator for selecting the best clipping threshold, which can achieve smaller accuracy loss than the traditional KL method.

[0059] ​In some embodiments, the difference between the mean of the absolute values of the quantized data and the mean of the absolute values of the data to be quantized can be the difference between the two means of the absolute values. Alternatively, the difference between the mean of the absolute values of the quantized data and the mean of the absolute values of the data to be quantized can also be the difference between the two means of the absolute values divided by the mean of the absolute values of the data to be quantized, and then taking the absolute value.

[0060] In some embodiments, after the best pair of clipping thresholds is selected, the selected pair of clipping thresholds can be used to quantize a set of data to be quantized to obtain quantized data, including: clipping the values in the set of data to be quantized that are greater than the clipping positive value to the clipping positive value, and clipping the values in the set of data to be quantized that are less than the clipping negative value to the clipping negative value; and then inputting the obtained quantized data into a neural network model for processing.

[0061] Figure 6 A flowchart of a method 600 for searching for clipping thresholds for symmetric quantization is shown, according to an embodiment of the present disclosure, the method 600 determines the best pair of clipping thresholds based on data to be quantized for quantization of the data.

[0062] At block 602, the mean of the absolute values of the data to be quantized and the maximum absolute value in the data to be quantized are determined, where the mean of the absolute values is the sum of the absolute values of all data in the data to be quantized divided by the number of elements, and in addition, the minimum mean difference is initialized, for example, the maximum value in a floating point number is initially set, and the search order i of the loop search is initialized (for example, initialized to 0). In some embodiments, the search order i can also be initialized to half of the total number of searches, that is, the search starts from the middle, which can improve the search efficiency. According to an embodiment of the present disclosure, one or more rounds of threshold search processes can be set, and each round of threshold search can have the same or different total number of searches. In some embodiments, the total number of searches of each round can be set to between 10 and 32. Generally, the more the total number of searches, the longer the search time spent, and the more accurate the clipping thresholds searched. However, when the total number of searches reaches a certain value, the search effect can no longer be substantially improved.

[0063] Next, a first round of coarse-grained clipping threshold search process is started. For example, Figure 7A An example diagram 700 for coarse-grained search for clipping thresholds for symmetric quantization is shown, according to an embodiment of the present disclosure. As shown, Figure 7A Ten candidate clipping thresholds can be determined in the data to be quantized (identified by the dashed lines in FIG. 7A), and these ten pairs of clipping thresholds are used in turn Figure 7A Figure 7A ​The quantization process is performed and the difference between the mean of the absolute values of the data before and after quantization is used to determine the optimal pair of clipping thresholds.

[0064] At block 604, it is determined whether the search order i is less than the total search number, i.e. whether the calculation of all pairs of clipping thresholds has been completed when each pair of clipping thresholds is selected in turn for quantization. If the search order i is less than the total search number, then at block 606, a pair of clipping thresholds is determined based on the current search order i, the pair of clipping thresholds being -absolute value maximum / total search number*(i+1), absolute value maximum / total search number*(i+1) respectively. At block 608, the data to be quantized is quantized using the pair of clipping thresholds to obtain corresponding quantized data Quant_data_i, and then at block 610, the difference Distance_i = abs(Quant_data_mean_i-Data_mean) / Data_mean between the mean of the absolute values of the quantized data Quant_data_mean_i and the mean of the absolute values of the data to be quantized Data_mean is calculated.

[0065] At block 612, it is determined whether the calculated difference Distance_i is less than the current minimum difference. If so, then at block 614, the calculated difference Distance_i is set as the current minimum difference and the clipping thresholds at which the difference is minimum are recorded, and then at block 616, the search order i is incremented (i.e. i++). If not, then at block 616, the search order i is directly incremented, i.e. the difference when the next pair of clipping thresholds is determined is continued. Next, the steps 604 to 616 are continued until the value of the search order i reaches the total search number, and then at block 618, the first round of the search process for the clipping thresholds is exited. As shown, after the first round of the search, it is determined that the difference corresponding to the clipping thresholds at the dotted line 770 is minimum. It can be seen that the process of the clipping threshold search is that the data to be quantized is quantized using a plurality of pairs of clipping thresholds, the group of quantized data having the minimum difference with the data to be quantized in terms of the mean of the absolute values is determined from a plurality of groups of quantized data, and then a pair of clipping thresholds corresponding to the group of quantized data is selected from the plurality of pairs of clipping thresholds. Figure 7A

[0066] ​Optionally, a second round of fine-grained search of the clipping threshold can be performed, which can also refer to the method 600, except that the second round of search is performed around a certain range (e.g., between a previous clipping threshold and a next clipping threshold of the selected clipping threshold 770) of the first round of best clipping threshold 770, which is a further refinement of the first round of search result. For example, the interval between each pair of clipping threshold in the second round of search can be (absolute maximum value * 2) / (total number of first round of search * total number of second round of search). Figure 7B A diagram 750 for fine-grained search of clipping threshold for symmetric quantization according to an embodiment of the disclosure is shown, which refers to the method 600. Figure 7B After the second round of search, the fine-grained best clipping thresholds are determined to be 772 and 778. By means of the two rounds of search, a more accurate clipping threshold can be obtained, which further reduces the precision loss caused by quantization.

[0067] Figure 8 A flowchart of a method 800 for iteratively searching for the best clipping threshold according to an embodiment of the disclosure is shown. In block 802, three pairs of clipping thresholds are determined, for example, the absolute maximum value absmax of all data in the data F x to be quantized can be determined, and the three pairs of clipping thresholds can be (-absmax / 2, absmax / 2), (-absmax*3 / 4, absmax*3 / 4) and (-absmax, absmax) respectively. In block 804, the data to be quantized is quantized using the three pairs of clipping thresholds respectively to obtain the quantized data F x The average of the corresponding absolute values F mean The minimum difference diff_min is then selected according to the formula In block 806, it is determined whether the minimum difference diff_min is less than a predetermined threshold set in advance. If not, in block 808, based on the selected pair of clipping thresholds (set the value corresponding to the minimum difference diff_min as the new absolute maximum value), the three pairs of clipping thresholds are re-determined, and the above process is repeated until the minimum difference diff_min is less than the predetermined threshold, then in block 810, the iteration process of the clipping threshold is exited. In some embodiments, in addition to the iteration stop condition that the minimum difference diff_min is less than the predetermined threshold, other iteration stop conditions can also be set, such as the maximum number of iterations, reaching a predetermined minimum interval, etc. In addition, although the method 800 of the present disclosure shows that the best pair of clipping thresholds is iteratively selected, it is also possible not to perform iteration, but only to perform once, and then directly take the pair of clipping thresholds corresponding to the minimum difference diff_min as the final clipping threshold. Figure 8 In block 806, it is determined whether the minimum difference diff_min is less than a predetermined threshold set in advance. If not, in block 808, based on the selected pair of clipping thresholds (set the value corresponding to the minimum difference diff_min as the new absolute maximum value), the three pairs of clipping thresholds are re-determined, and the above process is repeated until the minimum difference diff_min is less than the predetermined threshold, then in block 810, the iteration process of the clipping threshold is exited. In some embodiments, in addition to the iteration stop condition that the minimum difference diff_min is less than the predetermined threshold, other iteration stop conditions can also be set, such as the maximum number of iterations, reaching a predetermined minimum interval, etc. In addition, although the method 800 of the present disclosure shows that the best pair of clipping thresholds is iteratively selected, it is also possible not to perform iteration, but only to perform once, and then directly take the pair of clipping thresholds corresponding to the minimum difference diff_min as the final clipping threshold.​​​

[0068] In some embodiments, the quantization parameters when using each pair of truncation thresholds to quantize data can be determined by the following equations (1)-(3).

[0069]

[0070]

[0071]

[0072] Where p is the maximum absolute value in the data to be quantized, n represents the number of bits after quantization, S and f represent the quantization parameters, and ceil represents rounding up.

[0073] According to embodiments of this disclosure, by selecting p as absmax / 2, absmax*3 / 4, and absmax respectively, the quantization parameters S1, f1, S2, f2, S3, and f3 can be obtained, thereby obtaining the quantized data. Accordingly, after selecting a pair of cutoff thresholds, S and f corresponding to this pair of cutoff thresholds are directly taken as the quantization parameters of the data to be quantized.

[0074] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0075] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0076] Figure 9 A block diagram of an apparatus 900 for processing data according to an embodiment of the present disclosure is shown. Figure 9As shown, the apparatus 900 includes a data acquisition unit 910, a quantized data determination unit 920, and a truncation threshold selection unit 930. The data acquisition unit 910 acquires a set of data to be quantized for a machine learning model. The quantized data determination unit 920 determines multiple sets of quantized data by quantizing the set of data to be quantized using multiple pairs of truncation thresholds, wherein each pair of truncation thresholds includes a symmetrical truncation positive value and a truncation negative value. The truncation threshold selection unit 930 selects a pair of truncation thresholds from the multiple pairs of truncation thresholds to quantize the set of data to be quantized, based on the difference between the mean of the absolute values ​​of each set of quantized data and the mean of the absolute values ​​of the set of data to be quantized.

[0077] Furthermore, the data acquisition unit 910 to be quantized, the data determination unit 920 after quantization, and the truncation threshold selection unit 930 in the device 900 may also be configured to perform steps and / or actions according to various embodiments of the present disclosure.

[0078] It should be understood that the above-described device embodiments are merely illustrative, and the device disclosed herein can be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical functional division, and other division methods may be used in actual implementation. For example, multiple units, modules, or components may be combined, integrated into another system, or some features may be ignored or not executed.

[0079] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this disclosure can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0080] If the integrated unit / module is implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0081] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0082] In one embodiment, a computer-readable storage medium is disclosed having a computer program stored thereon, which, when executed, implements the methods according to various embodiments of the present disclosure.

[0083] In one embodiment, an artificial intelligence chip is also disclosed, which includes means for processing the aforementioned data.

[0084] In one embodiment, a board is also disclosed, which includes a storage device, an interface device, a controller, and the aforementioned artificial intelligence chip; wherein the artificial intelligence chip is connected to the storage device, the controller, and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the artificial intelligence chip and external devices; and the controller is used to monitor the status of the artificial intelligence chip.

[0085] Figure 10 A structural block diagram of a board 1000 according to an embodiment of the present disclosure is shown, with reference to Figure 10 In addition to the chips 1030-1 and 1030-2 (collectively referred to as chip 1030), the aforementioned board 1000 may also include other supporting components, including but not limited to: a storage device 1010, an interface device 1040, and a controller 1020. The interface device 1040 can be connected to an external device 1060. The storage device 1010 is connected to the artificial intelligence chip 1030 via a bus 1050 and is used for data storage. The storage device 1010 may include multiple sets of storage cells 1010-1 and 1010-2. Each set of storage cells is connected to the artificial intelligence chip via a bus 1050. It is understood that each set of storage cells may be DDR SDRAM (Double Data Rate SDRAM).

[0086] DDR can double the speed of SDRAM without increasing the clock frequency. DDR allows data to be read on both the rising and falling edges of the clock pulse. DDR is twice as fast as standard SDRAM. In one embodiment, the storage device may include four groups of storage cells. Each group of storage cells may include multiple DDR4 chips. In one embodiment, the AI ​​chip may internally include four 72-bit DDR4 controllers, of which 64 bits are used for data transmission and 8 bits are used for ECC verification. It can be understood that when DDR4-3200 chips are used in each group of storage cells, the theoretical data transmission bandwidth can reach 25600MB / s.

[0087] In one embodiment, each group of memory cells includes multiple Double Data Rate (DDR) synchronous dynamic random access memories (DRAMs) arranged in parallel. DDR can transfer data twice within one clock cycle. A controller for controlling the DDR is provided in the chip for controlling the data transfer and data storage of each memory cell.

[0088] The interface device is electrically connected to the artificial intelligence chip. The interface device is used to realize data transmission between the artificial intelligence chip and external devices (e.g., servers or computers). For example, in one embodiment, the interface device can be a standard PCIe interface. For instance, data to be processed is transferred from the server to the chip via a standard PCIe interface, realizing data transfer. Preferably, when using a PCIe 3.0 x 16 interface, the theoretical bandwidth can reach 16000 MB / s. In another embodiment, the interface device can also be other interfaces; this disclosure does not limit the specific form of the other interfaces mentioned above, as long as the interface unit can realize the switching function. Furthermore, the calculation results of the artificial intelligence chip are still transmitted back to the external device (e.g., the server) by the interface device.

[0089] The controller is electrically connected to the AI ​​chip. The controller monitors the state of the AI ​​chip. Specifically, the AI ​​chip and the controller can be electrically connected via an SPI interface. The controller may include a microcontroller (MCU). The AI ​​chip may include multiple processing chips, multiple processing cores, or multiple processing circuits, capable of driving multiple loads. Therefore, the AI ​​chip can operate in different states, such as high load and low load. The controller can regulate the operating states of the multiple processing chips, multiple processing cores, and / or multiple processing circuits within the AI ​​chip.

[0090] In one possible implementation, an electronic device is disclosed that includes the aforementioned artificial intelligence chip. The electronic device includes data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, dashcams, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and / or medical devices.

[0091] The means of transportation include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical equipment includes MRI scanners, ultrasound scanners and / or electrocardiographs.

[0092] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0093] The foregoing can be better understood in accordance with the following terms:

[0094] A1. A method for processing data, characterized in that it includes:

[0095] Obtain a set of data to be quantized for use in machine learning models;

[0096] Multiple sets of quantized data are determined by quantizing the set of data to be quantized using multiple pairs of truncation thresholds, each pair of thresholds including a symmetrical truncation positive value and a truncation negative value; and

[0097] Based on the difference between the mean absolute value of each group of quantized data and the mean absolute value of the group of data to be quantized, a pair of cutoff thresholds is selected from the multiple pairs of cutoff thresholds to be used for quantizing the group of data to be quantized.

[0098] A2. The method according to clause A1, characterized in that determining multiple sets of quantized data includes:

[0099] Determine the maximum absolute value of all data in the set of data to be quantized; and

[0100] The multiple truncation thresholds are determined based on the maximum absolute value.

[0101] A3. The method according to clause A2, characterized in that determining multiple sets of quantized data further includes:

[0102] Based on the maximum absolute value, the predetermined total number of searches, and the current search order, determine the first truncation positive value;

[0103] The first set of quantized data is determined by quantizing the set of data to be quantized using a first pair of cutoff thresholds, wherein the first pair of cutoff thresholds includes a first cutoff positive value and a first cutoff negative value opposite to the first cutoff positive value; and

[0104] Determine the first difference between the mean of the absolute values ​​of the first set of quantized data and the mean of the absolute values ​​of the set of data to be quantized.

[0105] A4. The method described according to clause A3, characterized in that determining the multiple sets of quantized data further includes:

[0106] Increment the current search order;

[0107] The second truncation positive value is determined based on the maximum absolute value, the predetermined total number of searches, and the current search order;

[0108] A second set of quantized data is determined by quantizing the set of data to be quantized using a second pair of cutoff thresholds, wherein the second pair of cutoff thresholds includes a second positive cutoff value and a second negative cutoff value that is opposite to the second positive cutoff value; and

[0109] Determine a second difference between the mean of the absolute values ​​of the second set of quantized data and the mean of the absolute values ​​of the set of data to be quantized.

[0110] A5. The method according to any one of clauses A1-A4, characterized in that selecting a pair of cutoff thresholds from the plurality of pairs of cutoff thresholds comprises:

[0111] Determine the set of quantized data that has the smallest difference in absolute mean from the set of data to be quantized among the multiple sets of quantized data; and

[0112] Select a pair of truncation thresholds that corresponds to the set of quantized data from the plurality of truncation thresholds.

[0113] A6. The method according to clause A5, characterized in that it further comprises:

[0114] Determine the cutoff search range associated with the selected pair of cutoff thresholds;

[0115] Determine new multiple pairs of truncation thresholds within the truncation search range;

[0116] By using the new multiple pairs of cutoff thresholds to quantize each set of data to be quantized, new sets of quantized data are determined; and

[0117] Based on the difference between the mean of the absolute values ​​of each group of quantized data in the new multi-group quantized data and the mean of the absolute values ​​of the group of data to be quantized, a new pair of cutoff thresholds is selected from the new multi-pair cutoff thresholds.

[0118] A7. The method according to clause A1, characterized in that determining multiple sets of quantized data includes:

[0119] Determine the maximum absolute value of all data in the set of data to be quantized;

[0120] Based on the maximum absolute value, three pairs of cutoff thresholds are determined. The first pair of cutoff thresholds comprises half of the maximum absolute value and its opposite; the second pair comprises three-quarters of the maximum absolute value and its opposite; and the third pair comprises the maximum absolute value and its opposite.

[0121] The three sets of quantized data are determined by quantizing the set of data to be quantized using three pairs of cutoff thresholds.

[0122] A8. The method according to clause A7, characterized in that selecting a pair of cutoff thresholds from the plurality of pairs of cutoff thresholds includes:

[0123] The following actions will be performed iteratively until the stopping condition is met:

[0124] Choose one pair of cutoff thresholds from the three pairs;

[0125] Determine whether the difference corresponding to the selected pair of cutoff thresholds is less than a predetermined threshold;

[0126] In response to the difference being less than a predetermined threshold, the iterative execution of the action is stopped; and

[0127] In response to the difference being greater than a predetermined threshold, three pairs of truncation thresholds are re-determined based on the selected pair of truncation thresholds.

[0128] A9. The method according to any one of clauses A1-A8, characterized in that the set of data to be quantized is a set of floating-point numbers in a neural network model, the method further comprising:

[0129] The set of data to be quantized is quantized using a selected pair of cutoff thresholds to obtain quantized data, wherein quantizing the set of data to be quantized includes: setting values ​​in the set of data to be quantized that are greater than a positive cutoff value as the positive cutoff value, and setting values ​​in the set of data to be quantized that are less than a negative cutoff value as the negative cutoff value; and

[0130] The obtained quantized data is input into the neural network model for processing.

[0131] A10. An apparatus for processing data, characterized in that it comprises:

[0132] The data acquisition unit is used to acquire a set of data to be quantized for use in machine learning models.

[0133] The quantized data determination unit is used to determine multiple sets of quantized data by quantizing the set of data to be quantized using multiple pairs of truncation thresholds, wherein each pair of truncation thresholds includes a symmetrical truncation positive value and a truncation negative value; and

[0134] The truncation threshold selection unit is used to select a pair of truncation thresholds from the plurality of truncation thresholds based on the difference between the mean of the absolute values ​​of each group of quantized data in the plurality of quantized data and the mean of the absolute values ​​of the group of data to be quantized, so as to quantize the group of data to be quantized.

[0135] A11. The apparatus according to clause A10, characterized in that the quantized data determination unit comprises:

[0136] An absolute value maximum value determination unit is used to determine the absolute value maximum value of all data in the set of data to be quantized; and

[0137] A multi-pair truncation threshold determination unit is used to determine the multi-pair truncation thresholds based on the maximum absolute value.

[0138] A12. The apparatus according to clause A11, characterized in that the quantized data determination unit further comprises:

[0139] The first truncation positive value determination unit is used to determine the first truncation positive value based on the maximum absolute value, the predetermined total number of searches, and the current search order;

[0140] The first set of quantized data determination unit is used to determine the first set of quantized data by quantizing the set of data to be quantized using a first pair of truncation thresholds, wherein the first pair of truncation thresholds includes a first truncation positive value and a first truncation negative value opposite to the first truncation positive value; and

[0141] The first difference determination unit is used to determine the first difference between the mean of the absolute values ​​of the first group of quantized data and the mean of the absolute values ​​of the group of data to be quantized.

[0142] A13. The apparatus according to clause A12, characterized in that the quantized data determination unit further comprises:

[0143] An incrementing unit is used to increment the current search order;

[0144] The second truncation positive value determination unit is used to determine the second truncation positive value based on the maximum absolute value, the predetermined total number of searches, and the current search order.

[0145] The second set of quantized data determination units is used to determine the second set of quantized data by quantizing the set of data to be quantized using a second pair of truncation thresholds, wherein the second pair of truncation thresholds includes a second truncation positive value and a second truncation negative value opposite to the second truncation positive value; and

[0146] The second difference determination unit is used to determine a second difference between the mean of the absolute values ​​of the second group of quantized data and the mean of the absolute values ​​of the group of data to be quantized.

[0147] A14. The apparatus according to any one of clauses A10-A13, characterized in that the cutoff threshold selection unit comprises:

[0148] A minimum difference determination unit is used to determine the set of quantized data that has the smallest difference in absolute value from the set of data to be quantized among the multiple sets of quantized data; and

[0149] The second truncation threshold selection unit is used to select a pair of truncation thresholds from the plurality of pairs of truncation thresholds that correspond to the set of quantized data.

[0150] A15. The apparatus according to clause A14, characterized in that it further comprises:

[0151] A truncation search range determination unit is used to determine the truncation search range associated with the selected pair of truncation thresholds;

[0152] A new multi-pair truncation threshold determination unit is used to determine new multi-pair truncation thresholds within the truncation search range;

[0153] The second quantized data determination unit is used to determine new sets of quantized data by quantizing the set of data to be quantized using the new multiple pairs of truncation thresholds; and

[0154] The third truncation threshold selection unit is used to select a new pair of truncation thresholds from the new multiple pairs of truncation thresholds based on the difference between the mean of the absolute values ​​of each group of quantized data in the new multiple groups of quantized data and the mean of the absolute values ​​of the group of data to be quantized.

[0155] A16. The apparatus according to clause A10, characterized in that the quantized data determination unit comprises:

[0156] An absolute value maximum value determination unit is used to determine the maximum absolute value of all data in the set of data to be quantized.

[0157] A three-pair truncation threshold determination unit is configured to determine three pairs of truncation thresholds based on the maximum absolute value. The first pair of truncation thresholds comprises half of the maximum absolute value and its opposite; the second pair comprises three-quarters of the maximum absolute value and its opposite; and the third pair comprises the maximum absolute value and its opposite.

[0158] The three-group quantized data determination unit is used to determine the three groups of quantized data by quantizing the group of data to be quantized by using three pairs of truncation thresholds respectively.

[0159] A17. The apparatus according to clause A16, characterized in that the cutoff threshold selection unit comprises:

[0160] An iterative unit is used to iteratively execute the following actions until a stopping condition is met:

[0161] Choose one pair of cutoff thresholds from the three pairs;

[0162] Determine whether the difference corresponding to the selected pair of cutoff thresholds is less than a predetermined threshold;

[0163] In response to the difference being less than a predetermined threshold, the iterative execution of the action is stopped; and

[0164] In response to the difference being greater than a predetermined threshold, three pairs of truncation thresholds are re-determined based on the selected pair of truncation thresholds.

[0165] A18. The apparatus according to any one of clauses A10-A17, characterized in that the set of data to be quantized is a set of floating-point numbers in a neural network model, the apparatus further comprising:

[0166] A data quantization unit is configured to quantize the set of data to be quantized using a selected pair of cutoff thresholds to obtain quantized data, wherein quantizing the set of data to be quantized includes: setting values ​​in the set of data to be quantized that are greater than a positive cutoff value as the positive cutoff value, and setting values ​​in the set of data to be quantized that are less than a negative cutoff value as the negative cutoff value; and

[0167] The data input unit is used to input the obtained quantized data into the neural network model for processing.

[0168] A19. A computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed, implements the method according to any one of clauses A1-A9.

[0169] A20. An artificial intelligence chip, characterized in that the chip includes means for processing data according to any one of clauses A10-A18.

[0170] A21. An electronic device, characterized in that the electronic device includes an artificial intelligence chip as described in clause A20.

[0171] A22. A circuit board, characterized in that the circuit board comprises: a storage device, an interface device, a control device, and an artificial intelligence chip as described in clause A20;

[0172] The artificial intelligence chip is connected to the storage device, the control device, and the interface device.

[0173] The storage device is used to store data;

[0174] The interface device is used to realize data transmission between the artificial intelligence chip and external devices; and

[0175] The controller is used to monitor the state of the artificial intelligence chip.

[0176] A23. The circuit board as described in clause A22, characterized in that,

[0177] The storage device includes: multiple sets of storage units, each set of storage units being connected to the artificial intelligence chip via a bus, and the storage units being DDR SDRAM;

[0178] The chip includes: a DDR controller for controlling data transmission and data storage of each memory cell;

[0179] The interface device is a standard PCIe interface.

[0180] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this disclosure, and on the specific implementation methods and application scope of this disclosure, are all within the scope of protection of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A method for processing data, characterized by, comprising: obtaining a set of data to be quantized for a machine learning model; determining a plurality of sets of quantized data by respectively quantizing the set of data to be quantized using a plurality of pairs of clipping thresholds, each pair of clipping thresholds comprising a symmetric positive clipping threshold and a symmetric negative clipping threshold; and selecting a pair of clipping thresholds from the plurality of pairs of clipping thresholds for quantizing the set of data to be quantized based on a difference between a mean of absolute values of each set of quantized data in the plurality of sets of quantized data and a mean of absolute values of the set of data to be quantized; wherein the machine learning model is used for one or more of: image recognition, speech recognition, or natural language processing.

2. The method of claim 1, wherein, Determining a plurality of sets of quantized data comprises: determining an absolute value maximum of all data in the set of data to be quantized; and determining the plurality of pairs of clipping thresholds based on the absolute value maximum.

3. The method of claim 2, wherein, Determining a plurality of sets of quantized data further comprises: determining a first positive clipping threshold based on the absolute value maximum, a predetermined total number of searches, and a current search order; determining a first set of quantized data by quantizing the set of data to be quantized using a first pair of clipping thresholds comprising the first positive clipping threshold and a first negative clipping threshold opposite the first positive clipping threshold; and determining a first difference between a mean of absolute values of the first set of quantized data and the mean of absolute values of the set of data to be quantized.

4. The method of claim 3, wherein, Determining a plurality of sets of quantized data further comprises: incrementing the current search order; determining a second positive clipping threshold based on the absolute value maximum, the predetermined total number of searches, and the current search order; determining a second set of quantized data by quantizing the set of data to be quantized using a second pair of clipping thresholds comprising the second positive clipping threshold and a second negative clipping threshold opposite the second positive clipping threshold; and determining a second difference between a mean of absolute values of the second set of quantized data and the mean of absolute values of the set of data to be quantized.

5. The method according to any one of claims 1-4, characterized in that, Selecting a pair of clipping thresholds from the plurality of pairs of clipping thresholds comprises: determining a set of quantized data in the plurality of sets of quantized data that has a smallest difference in the mean of absolute values from the set of data to be quantized; and selecting a pair of clipping thresholds from the plurality of pairs of clipping thresholds corresponding to the set of quantized data.

6. The method of claim 5, wherein, Further comprising: determining a clipping search range associated with the selected pair of clipping thresholds; determining new pairs of clipping thresholds within the clipping search range; determining new sets of quantized data by respectively quantizing the set of data to be quantized using the new pairs of clipping thresholds; and selecting a new pair of clipping thresholds from the new pairs of clipping thresholds based on a difference between a mean of absolute values of each set of quantized data in the new sets of quantized data and the mean of absolute values of the set of data to be quantized.

7. The method of claim 1, wherein, Determining a plurality of sets of quantized data by respectively quantizing the set of data to be quantized using a plurality of pairs of clipping thresholds comprises: determining an absolute value maximum of all data in the set of data to be quantized; determining three pairs of clipping thresholds based on the maximum absolute value, a first pair of clipping thresholds of the three pairs of clipping thresholds including one half of the maximum absolute value and an opposite number thereof, a second pair of clipping thresholds of the three pairs of clipping thresholds including three quarters of the maximum absolute value and an opposite number thereof, and a third pair of clipping thresholds of the three pairs of clipping thresholds including the maximum absolute value and an opposite number thereof; and determining three sets of quantized data by quantizing the set of data to be quantized using the three pairs of clipping thresholds respectively.

8. The method of claim 7, wherein, selecting a pair of clipping thresholds from the pairs of clipping thresholds comprises: iteratively performing the following actions until a stop condition is satisfied: selecting a pair of clipping thresholds from the three pairs of clipping thresholds; determining whether a difference corresponding to the selected pair of clipping thresholds is less than a predetermined threshold; in response to the difference being less than the predetermined threshold, stopping the iterative performance of the actions; and in response to the difference being greater than the predetermined threshold, re-determining the three pairs of clipping thresholds based on the selected pair of clipping thresholds.

9. The method according to any one of claims 1-8, characterized in that, the set of data to be quantized is a set of floating-point numbers in a neural network model, the method further comprising: quantizing the set of data to be quantized using the selected pair of clipping thresholds to obtain quantized data, wherein quantizing the set of data to be quantized comprises: setting a value greater than a clipping positive value in the set of data to be quantized to the clipping positive value, and setting a value less than a clipping negative value in the set of data to be quantized to the clipping negative value; and inputting the obtained quantized data to the neural network model for processing.

10. An apparatus for processing data, characterized by comprises: a data to be quantized acquisition unit, configured to acquire a set of data to be quantized for a machine learning model; a quantized data determination unit, configured to determine a plurality of sets of quantized data by quantizing the set of data to be quantized using a plurality of pairs of clipping thresholds respectively, each pair of clipping thresholds of the plurality of pairs of clipping thresholds comprising a symmetric clipping positive value and a clipping negative value; and a clipping threshold selection unit, configured to select a pair of clipping thresholds from the pairs of clipping thresholds based on a difference between a mean value of absolute values of each set of quantized data in the plurality of sets of quantized data and a mean value of absolute values of the set of data to be quantized, for quantizing the set of data to be quantized; wherein the machine learning model is used for one or more of: image recognition, speech recognition, or natural language processing.

11. The apparatus of claim 10, wherein, the quantized data determination unit comprises: an absolute value maximum determination unit, configured to determine a maximum absolute value of all data in the set of data to be quantized; and a plurality of pairs of clipping threshold determination unit, configured to determine the plurality of pairs of clipping thresholds based on the maximum absolute value.

12. The apparatus of claim 11, wherein, the quantized data determination unit further comprises: a first clipping positive value determination unit, configured to determine a first clipping positive value based on the maximum absolute value, a predetermined total number of searches, and a current search order; a first set of quantized data determination unit, configured to determine a first set of quantized data by quantizing the set of data to be quantized using a first pair of clipping thresholds, the first pair of clipping thresholds comprising the first clipping positive value and a first clipping negative value opposite to the first clipping positive value; and a second set of quantized data determination unit, configured to determine a second set of quantized data by quantizing the set of data to be quantized using a second pair of clipping thresholds, the second pair of clipping thresholds comprising a second clipping positive value and a second clipping negative value opposite to the second clipping positive value. a first difference determination unit configured to determine a first difference between a mean of absolute values of the first set of quantized data and a mean of absolute values of the set of data to be quantized.

13. The apparatus of claim 12, wherein, The quantized data determination unit further includes: an increment unit configured to increment the current search order; a second truncated positive value determination unit configured to determine a second truncated positive value based on the maximum absolute value, the predetermined total number of searches, and the current search order; a second set of quantized data determination unit configured to determine a second set of quantized data by quantizing the set of data to be quantized using a second pair of truncated thresholds, the second pair of truncated thresholds including the second truncated positive value and a second truncated negative value opposite the second truncated positive value; and a second difference determination unit configured to determine a second difference between a mean of absolute values of the second set of quantized data and the mean of absolute values of the set of data to be quantized.

14. The apparatus of any one of claims 10-13, wherein, The truncated threshold selection unit includes: a minimum difference determination unit configured to determine a set of quantized data from the multiple sets of quantized data that has a minimum difference from the set of data to be quantized in terms of the mean of absolute values; and a second truncated threshold selection unit configured to select a pair of truncated thresholds from the multiple pairs of truncated thresholds that corresponds to the set of quantized data.

15. The apparatus of claim 14, wherein, The truncated threshold selection unit further includes: a truncated search range determination unit configured to determine a truncated search range associated with the selected pair of truncated thresholds; a new multiple pairs of truncated threshold determination unit configured to determine a new multiple pairs of truncated thresholds within the truncated search range; a second quantized data determination unit configured to determine a new multiple sets of quantized data by quantizing the set of data to be quantized using the new multiple pairs of truncated thresholds, respectively; and a third truncated threshold selection unit configured to select a new pair of truncated thresholds from the new multiple pairs of truncated thresholds based on a difference between a mean of absolute values of each set of quantized data from the new multiple sets of quantized data and the mean of absolute values of the set of data to be quantized. The quantized data determination unit includes:

16. The apparatus of claim 10, wherein, a maximum absolute value determination unit configured to determine a maximum absolute value of all data in the set of data to be quantized; a multiple pairs of truncated threshold determination unit configured to determine a multiple pairs of truncated thresholds based on the maximum absolute value, a first pair of truncated thresholds from the multiple pairs of truncated thresholds including half of the maximum absolute value and an opposite number thereof, a second pair of truncated thresholds from the multiple pairs of truncated thresholds including three quarters of the maximum absolute value and an opposite number thereof, and a third pair of truncated thresholds from the multiple pairs of truncated thresholds including the maximum absolute value and an opposite number thereof; and a multiple sets of quantized data determination unit configured to determine a multiple sets of quantized data by quantizing the set of data to be quantized using the multiple pairs of truncated thresholds, respectively. The truncated threshold selection unit includes:

17. The apparatus of claim 16, wherein, an iteration unit configured to iteratively perform the following actions until a stop condition is satisfied: selecting a pair of truncated thresholds from the multiple pairs of truncated thresholds; determining whether a difference corresponding to the selected pair of truncated thresholds is less than a predetermined threshold; in response to the difference being less than the predetermined threshold, stopping the iterative performance of the actions; and ​ In response to the difference being greater than a predetermined threshold, re-determine three pairs of clipping thresholds based on the selected pair of clipping thresholds.

18. The apparatus of any one of claims 10-17, wherein, The set of data to be quantized is a set of floating-point numbers in a neural network model, and the apparatus further comprises: a data quantization unit configured to quantize the set of data to be quantized using the selected pair of clipping thresholds to obtain quantized data, wherein quantizing the set of data to be quantized comprises setting a value greater than a clipping positive value in the set of data to be quantized to the clipping positive value, and setting a value less than a clipping negative value in the set of data to be quantized to the clipping negative value; and a data input unit configured to input the obtained quantized data to the neural network model for processing.

19. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the program is executed to implement the method according to any one of claims 1-9.

20. An artificial intelligence chip, comprising: The chip comprises the apparatus for processing data according to any one of claims 10-18.

21. An electronic device, comprising: The electronic device comprises the artificial intelligence chip according to claim 20.

22. A board card, characterized by The board card comprises a memory device, an interface device and a control device, and the artificial intelligence chip according to claim 20; The artificial intelligence chip is connected with the memory device, the control device and the interface device; The memory device is configured to store data; The interface device is configured to realize data transmission between the artificial intelligence chip and an external device; and The control device is configured to monitor a state of the artificial intelligence chip.

23. The board card according to claim 22, characterized in that, The memory device comprises a plurality of groups of storage units, each group of storage units being connected with the artificial intelligence chip through a bus, and the storage units are DDR SDRAMs; The chip comprises a DDR controller configured to control data transmission and data storage of each storage unit; The interface device is a standard PCIE interface.

Images