Method for determining quantization parameters of a neural network and related products

Abstract

A method for determining quantization parameters of a neural network and related products. The related products include a board card, the board card including: a storage device (390), an interface device (391), a control device (392), and an artificial intelligence chip (389); wherein the artificial intelligence chip (389) is connected to the storage device (390), the control device (392), and the interface device (391) respectively; the storage device (390) is used to store data; the interface device (391) is used to realize data transmission between the artificial intelligence chip (389) and an external device; and the control device (392) is used to monitor the status of the artificial intelligence chip (389). The board card can be used to perform artificial intelligence operations.

Description

Technical Field

[0001] The embodiments disclosed herein relate to a method for determining quantization parameters of a neural network and related products. Background Art

[0002] A neural network (NN) is a mathematical or computational model that mimics the structure and function of biological neural networks. Through training with sample data, NNs continuously adjust network weights and thresholds, causing the error function to decrease along the negative gradient, approaching the desired output. It is a widely used recognition and classification model, often used for function approximation, model recognition and classification, data compression, and time series prediction.

[0003] In actual applications, neural network data often uses 32 bits. Existing neural network data occupies more bits. Although it ensures accuracy, it requires higher storage space and processing bandwidth, which increases the cost. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, the present disclosure proposes a method for adjusting data bit width and related products.

[0005] To achieve the above objectives, the present disclosure provides a method for adjusting data bit width, characterized in that the method includes:

[0006] Acquire a data bit width for performing a quantization process on the data to be quantized, wherein the data bit width indicates a bit width of quantized data of the data to be quantized after the quantization process;

[0007] Based on the data bit width, performing quantization processing on a set of data to be quantized to convert the set of data to be quantized into a set of quantized data, wherein the set of quantized data has the data bit width;

[0008] comparing the set of data to be quantized and the set of quantized data to determine a quantization error associated with the data bit width; and

[0009] The data bit width is adjusted based on the determined quantization error.

[0010] To achieve the above objectives, the present disclosure provides a device for adjusting data bit width, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the above method when executing the computer program.

[0011] To achieve the above objectives, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-described method when executed by a processor.

[0012] To achieve the above objectives, the present disclosure provides a device for adjusting data bit width, the device comprising:

[0013] An acquiring unit, configured to acquire a data bit width for performing a quantization process on the data to be quantized, wherein the data bit width indicates a bit width of quantized data after the data to be quantized undergoes the quantization process;

[0014] a quantization unit, configured to perform quantization processing on a set of to-be-quantized data based on the data bit width, so as to convert the set of to-be-quantized data into a set of quantized data, wherein the set of quantized data has the data bit width;

[0015] a determining unit, configured to compare the set of data to be quantized with the set of quantized data to determine a quantization error associated with the data bit width; and

[0016] An adjusting unit is configured to adjust the data bit width based on the determined quantization error.

[0017] During the neural network operation process, the technical solution disclosed in this disclosure is used to determine the data bit width during quantization. This data bit width is used by the artificial intelligence processor to quantize the data in the neural network operation process and convert high-precision data into low-precision fixed-point numbers, which can reduce the total space size of data storage involved in the neural network operation process. For example, converting float32 to fix8 can reduce the model parameters by 4 times. Since the data storage space becomes smaller, the neural network uses a smaller space when deployed, allowing the on-chip memory on the artificial intelligence processor chip to accommodate more data, reducing the memory access data of the artificial intelligence processor chip and improving computing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present disclosure, and are not intended to limit the present disclosure.

[0019] Figure 1 Schematic diagram of neural network structure;

[0020] Figure 2 A flow chart of a method for determining quantization parameters of a neural network proposed in the present disclosure;

[0021] Figure 3 It is a schematic diagram of symmetrical fixed-point number representation;

[0022] Figure 4 A schematic diagram showing fixed-point numbers with offset introduced;

[0023] Figure 5AThis is one of the curves of the change range of the weight data of the neural network during the training process;

[0024] Figure 5B This is the second graph of the change range of the weight data of the neural network during the training process;

[0025] Figure 6 One of the flow charts of the method for determining the target iteration interval;

[0026] Figure 7 Flowchart 2 of the method for determining the target iteration interval;

[0027] Figure 8A Flowchart 3 of the method for determining the target iteration interval;

[0028] Figure 8B A flow chart showing a method for adjusting data bit width according to an embodiment of the present disclosure is shown;

[0029] Figure 8C A flow chart showing a method for adjusting data bit width according to another embodiment of the present disclosure is shown;

[0030] Figure 8D A flow chart showing a method for adjusting data bit width in another embodiment of the present disclosure is shown;

[0031] Figure 8E A flow chart showing a method for adjusting data bit width in another embodiment of the present disclosure is shown;

[0032] Figure 8F A flowchart showing a neural network quantization method according to an embodiment of the present disclosure;

[0033] Figure 8G A flowchart showing a neural network quantization method according to an embodiment of the present disclosure;

[0034] Figure 8H A flowchart showing a neural network quantization method according to an embodiment of the present disclosure;

[0035] Figure 8I A flowchart showing a neural network quantization method according to an embodiment of the present disclosure;

[0036] Figure 8J A flowchart showing a neural network quantization method according to an embodiment of the present disclosure;

[0037] Figure 8K A flowchart showing a method for adjusting a quantization parameter according to an embodiment of the present disclosure is shown;

[0038] Figure 8L A diagram showing a changing trend of the data to be quantified during a calculation process according to an embodiment of the present disclosure;

[0039] Figure 8MA flow chart illustrating a method for determining a target iteration interval in a parameter adjustment method according to an embodiment of the present disclosure;

[0040] Figure 8N A flow chart showing a method for determining a variation range of a point position according to an embodiment of the present disclosure;

[0041] Figure 8O A flow chart showing a method for determining a second mean value in one embodiment of the present disclosure;

[0042] Figure 8P A flow chart showing a method for determining a second mean value in another embodiment of the present disclosure;

[0043] Figure 8Q A flowchart showing a quantization parameter adjustment method according to another embodiment of the present disclosure;

[0044] Figure 8R A flowchart showing a quantization parameter adjustment method according to an embodiment of the present disclosure is provided;

[0045] Figure 8S A flow chart illustrating a method for determining a target iteration interval in a parameter adjustment method according to another embodiment of the present disclosure;

[0046] Figure 8T A flow chart illustrating a method for determining a target iteration interval in a parameter adjustment method according to another embodiment of the present disclosure;

[0047] Figure 8U A flowchart showing a quantization parameter adjustment method according to another embodiment of the present disclosure;

[0048] Figure 8V A flowchart showing a method for adjusting a quantization parameter according to another embodiment of the present disclosure is shown;

[0049] Figure 9 A block diagram of the hardware configuration of a device for determining quantization parameters of a neural network proposed in the present disclosure;

[0050] Figure 10 A schematic diagram of the application of the device for determining the quantization parameters of a neural network proposed in this disclosure to an artificial intelligence processor chip;

[0051] Figure 11 A functional block diagram of a device for determining quantization parameters of a neural network proposed in the present disclosure;

[0052] Figure 12 This is a structural block diagram of the board card according to an embodiment of the present disclosure.

[0053] In order to solve the problem of neural network quantization, we proposed the following solution (201910505239.7) including the following Figures 2-1 to 2-31 .

[0054] Figure 2-1 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0055] Figure 2-2 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0056] Figure 2-3 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0057] Figure 2-4 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0058] Figure 2-5 A schematic diagram showing the correspondence between data before and after quantization when the quantization parameter does not include an offset in the neural network quantization method according to an embodiment of the present disclosure.

[0059] Figure 2-6 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0060] Figure 2-7 A schematic diagram showing the correspondence between data before and after quantization when the quantization parameter includes an offset in the neural network quantization method according to an embodiment of the present disclosure.

[0061] Figure 2-8 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0062] Figure 2-9 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0063] Figure 2-10 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0064] Figure 2-11 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0065] Figure 2-12 A flowchart of a neural network quantization method according to an embodiment of the present disclosure is shown.

[0066] Figure 2-1 3 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0067] Figure 2-1 4 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0068] Figure 2-1 5 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0069] Figure 2-1 6 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0070] Figure 2-1 FIG7 shows a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0071] Figure 2-1 FIG8 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0072] Figure 2-1 FIG9 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0073] Figure 2-2 FIG0 shows a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0074] Figure 2-2 1 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0075] Figure 2-2 2 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0076] Figure 2-2 3 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0077] Figure 2-2 4 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0078] Figure 2-2 5 is a flowchart of a neural network quantization method according to an embodiment of the present disclosure.

[0079] Figure 2-2 6 is a schematic diagram of a neural network quantization device according to an embodiment of the present disclosure.

[0080] Figure 2-2 7 is a schematic diagram of a neural network quantization device according to an embodiment of the present disclosure.

[0081] Figure 2-2 FIG8 is a schematic diagram of a neural network quantization device according to an embodiment of the present disclosure.

[0082] Figure 2-2 FIG9 is a schematic diagram of a neural network quantization device according to an embodiment of the present disclosure.

[0083] Figure 2-3 FIG0 shows a schematic diagram of a neural network quantization device according to an embodiment of the present disclosure.

[0084] Figure 2-3 1 shows a structural block diagram of a board according to an embodiment of the present disclosure.

[0085] In order to solve the problem of quantization parameter adjustment, we proposed the following solution (201910528537.8) including the following Figure 3-1 to Figure 3-25 .

[0086] Figure 3-1 A schematic diagram illustrating an application environment of a quantization parameter adjustment method according to an embodiment of the present disclosure;

[0087] Figure 3-2 A schematic diagram showing the correspondence between the data to be quantified and the quantized data according to an embodiment of the present disclosure;

[0088] Figure 3-3 A schematic diagram showing the conversion of data to be quantized according to an embodiment of the present disclosure is shown;

[0089] Figure 3-4 A flowchart showing a method for adjusting a quantization parameter according to an embodiment of the present disclosure is shown;

[0090] Figure 3-5 A diagram showing a changing trend of the data to be quantified during a calculation process according to an embodiment of the present disclosure;

[0091] Figure 3-6 A flow chart illustrating a method for determining a target iteration interval in a parameter adjustment method according to an embodiment of the present disclosure;

[0092] Figure 3-7 A flow chart showing a method for determining a variation range of a point position according to an embodiment of the present disclosure;

[0093] Figure 3-8 A flow chart showing a method for determining a second mean value in one embodiment of the present disclosure;

[0094] Figure 3-9 A flow chart showing a method for adjusting data bit width according to an embodiment of the present disclosure is shown;

[0095] Figure 3-10 A flow chart showing a method for adjusting data bit width according to another embodiment of the present disclosure is shown;

[0096] Figure 3-11 A flow chart showing a method for adjusting data bit width in another embodiment of the present disclosure is shown;

[0097] Figure 3-12 A flow chart showing a method for adjusting data bit width in another embodiment of the present disclosure is shown;

[0098] Figure 3-1 3 is a flowchart showing a method for determining a second mean value in another embodiment of the present disclosure;

[0099] Figure 3-1 4 is a flowchart showing a method for adjusting a quantization parameter according to another embodiment of the present disclosure;

[0100] Figure 3-1 5 is a flowchart showing a quantization parameter adjustment method according to an embodiment of the present disclosure;

[0101] Figure 3-1 6 is a flowchart showing a method for determining a target iteration interval in a parameter adjustment method according to another embodiment of the present disclosure;

[0102] Figure 3-1 7 is a flowchart showing a method for determining a target iteration interval in a parameter adjustment method according to another embodiment of the present disclosure;

[0103] Figure 3-1 8 is a flowchart showing a method for adjusting a quantization parameter according to another embodiment of the present disclosure;

[0104] Figure 3-1 9 is a flowchart of a quantization parameter adjustment method according to another embodiment of the present disclosure;

[0105] Figure 3-2 0 shows a structural block diagram of a quantization parameter adjustment device according to an embodiment of the present disclosure;

[0106] Figure 3-2 1 is a structural block diagram of a quantization parameter adjustment device according to an embodiment of the present disclosure;

[0107] Figure 3-2 2 shows a structural block diagram of a quantization parameter adjustment device according to an embodiment of the present disclosure;

[0108] Figure 3-2 3 shows a structural block diagram of a quantization parameter adjustment device according to an embodiment of the present disclosure;

[0109] Figure 3-2 4 shows a structural block diagram of a quantization parameter adjustment device according to an embodiment of the present disclosure;

[0110] Figure 3-2 5 shows a structural block diagram of a board according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0111] The following will clearly and completely describe the technical solutions in the embodiments of this disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this disclosure, not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0112] It should be understood that the terms "first," "second," "third," and "fourth," etc. in the terms, specifications, and drawings of this disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprising" used in the specifications and terms of this disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0113] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. As used in this disclosure and in the Terms and Conditions, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" as used in this disclosure and in the Terms and Conditions refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0114] As used in this specification and terms and conditions, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0115] Definition of technical terms:

[0116] Floating-point numbers: The IEEE floating-point standard represents a number in the form of V = (-1)^sign*mantissa*2^E. Sign is the sign bit, with 0 representing a positive number and 1 representing a negative number. E represents the exponent, which weights the floating-point number to the power of 2 (which may be a negative power). Mantissa represents the mantissa, a binary fraction in the range of 1 to 2-ε, or 0-ε. The representation of a floating-point number in a computer is divided into three fields, which are used to encode these fields:

[0117] (1) A single sign bit s directly encodes the sign s.

[0118] (2) The k-bit exponent field encodes the exponent, exp = e(k-1)...e(1)e(0).

[0119] (3) The n-bit decimal field mantissa encodes the mantissa. However, the encoding result depends on whether the exponent stage is all zeros.

[0120] Fixed-point numbers are composed of three parts: a shared exponent, a sign, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign indicates the positive or negative value of the fixed-point number. The mantissa determines the number of significant digits in the fixed-point number, i.e., its precision. Taking an 8-bit fixed-point number as an example, its numerical calculation method is:

[0121] value=(-1) sign ×(mantissa)×2 (exponent-127)

[0122] Binary decimals: Any decimal number can be expressed using the formula ∑j*10 i For example, the decimal number 12.34 is expressed as: 12.34 = 1*10 1 +2*10 0 +3*10 -1 +4*10 -2 , the left side of the decimal point counts as a positive power of 10, and the right side of the decimal point counts as a negative power of 10. Similarly, binary decimals can also be expressed in this way, with the left side of the decimal point counting as a positive power of 2, and the right side of the decimal point counting as a negative power of 2. The decimal number 5.75 can be represented by the binary number 10111, which is 5.75=1*2 2 +0*2 1 +1*2 0 +1*2 -1 +1*2 -2 .

[0123] Overflow: In fixed-point arithmetic, numbers can be represented within a certain range. During an operation, if the value of a number exceeds the range that can be represented by a fixed-point number, it is called "overflow."

[0124] KL (Kullback-Leibler) divergence: also known as relative entropy, information divergence, or information gain. KL divergence measures the asymmetry between two probability distributions P and Q. It measures the average number of extra bits required to encode samples from P using a code based on Q. Typically, P represents the true distribution of the data, and Q represents a theoretical distribution, a model distribution, or an approximation of P.

[0125] Data bit width: how many bits are used to represent the data.

[0126] Quantization: The process of converting high-precision numbers previously expressed in 32-bit or 64-bit into fixed-point numbers that occupy less memory space. The process of converting high-precision numbers to fixed-point numbers will cause a certain loss in accuracy.

[0127] The following describes in detail the specific implementation of a method for determining quantization parameters of a neural network and related products provided by the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0128] A neural network (NN) is a mathematical model that mimics the structure and function of biological neural networks. Neural networks perform computations by connecting a large number of neurons. Therefore, a neural network is a computational model composed of a large number of interconnected nodes (or "neurons"). Each node represents a specific output function, called an activation function. Each connection between two neurons represents a weighted value for the signal passing through that connection, called a weight, which acts as the neural network's memory. The output of a neural network varies depending on the connections between neurons, the weights, and the activation function. In a neural network, a neuron is the basic unit. It receives a certain number of inputs and a bias, and when a signal (value) arrives, it is multiplied by a weight. A connection connects a neuron to another neuron in another layer or within the same layer, and the connection is associated with a weight. Furthermore, the bias is an additional input to a neuron that is always 1 and has its own connection weight. This ensures that a neuron will activate even if all inputs are empty (all 0s).

[0129] In practice, if a nonlinear function isn't applied to the neurons in a neural network, the neural network is simply a linear function and is no more powerful than a single neuron. If the output of a neural network is set between 0 and 1, for example, in the example of identifying a cat or dog, an output close to 0 can be considered a cat, and an output close to 1 can be considered a dog. To achieve this goal, activation functions, such as the sigmoid activation function, are introduced into the neural network. All you need to know about this activation function is that its return value is a number between 0 and 1. Therefore, the activation function is used to introduce nonlinearity into the neural network, narrowing the results of the neural network's calculations to a smaller range. In reality, how the activation function is expressed is not important; what matters is that a nonlinear function is parameterized through weights, and the nonlinear function can be modified by changing these weights.

[0130] like Figure 1 As shown in the figure, it is a schematic diagram of the neural network structure. Figure 1 The neural network shown in the figure includes three layers: input layer, hidden layer and output layer. Figure 1The hidden layers shown are 5 layers. The leftmost layer of the neural network is called the input layer, and the neurons in the input layer are called input neurons. The input layer is the first layer in the neural network, which receives the required input signals (values) and passes them to the next layer. It generally does not operate on the input signals (values) and has no associated weights and biases. Figure 1 In the neural network shown, there are 4 input signals x1, x2, x3, x4.

[0131] The hidden layer contains neurons (nodes) that are used to apply different transformations to the input data. Figure 1 The neural network shown has five hidden layers. The first hidden layer has four neurons (nodes), the second layer has five neurons, the third layer has six neurons, the fourth layer has four neurons, and the fifth layer has three neurons. Finally, the hidden layers pass the calculated values of the neurons to the output layer. Figure 1 The neural network shown fully connects each neuron in the five hidden layers, meaning that every neuron in each hidden layer is connected to every neuron in the next layer. It should be noted that not every hidden layer in a neural network is fully connected.

[0132] Figure 1 The rightmost layer of the neural network is called the output layer, and the neurons in the output layer are called output neurons. The output layer receives the output from the last hidden layer. Figure 1 In the neural network shown, the output layer has 3 neurons and 3 output signals y1, y2, and y3.

[0133] In practical applications, a large amount of sample data (including input and output) is given in advance to train the initial neural network. After the training is completed, a trained neural network is obtained. This neural network can give a correct output for the input of the future real environment.

[0134] Before we begin discussing the training of a neural network, we need to define the loss function. The loss function is a function that measures the performance of a neural network in performing a specific task. In some embodiments, the loss function can be obtained as follows: during the training of a neural network, for each sample data, it is passed along the neural network to obtain an output value, and then the output value is subtracted from the expected value and then squared. The loss function calculated in this way is the distance between the predicted value and the true value, and the purpose of training the neural network is to reduce the value of this distance or loss function. In some embodiments, the loss function can be expressed as:

[0135]

[0136] In the above formula, y represents the expected value, Refers to the actual result obtained by the neural network for each sample data in the sample data set, and i is the index of each sample data in the sample data set. Represents the expected value y and the actual result The error value between . m is the number of sample data in the sample data set. Let’s take cat and dog identification as an example. There is a data set consisting of pictures of cats and dogs. If the picture is of a dog, the corresponding label is 1, and if the picture is of a cat, the corresponding label is 0. This label corresponds to the expected value y in the above formula. When passing each sample picture to the neural network, we actually want to obtain the recognition result through the neural network. In order to calculate the loss function, we must traverse each sample picture in the sample data set and obtain the actual result corresponding to each sample picture. Then calculate the loss function according to the above definition. If the loss function is large, it means that the neural network has not been trained well and the weights need to be further adjusted.

[0137] When training a neural network, the weights are randomly initialized. Obviously, an initialized neural network will not provide good results. During the training process, if we start with a very poor neural network, we can obtain a highly accurate network through training.

[0138] The training process of a neural network is divided into two stages. The first stage is the forward processing of the signal, from the input layer through the hidden layer, and finally to the output layer. The second stage is the backward propagation of the gradient, from the output layer to the hidden layer, and finally to the input layer. The weights and biases of each layer in the neural network are adjusted according to the gradient.

[0139] During forward processing, input values are fed into the neural network's input layer, and output values, known as predicted values, are obtained from the neural network's output layer. When input values are provided to the neural network's input layer, no operations are performed. Within the hidden layer, the second hidden layer receives the intermediate predicted values from the first hidden layer, performs computations and activations, and then passes the resulting intermediate predicted values to the next hidden layer. The same operations are performed in subsequent layers, ultimately resulting in the output value at the neural network's output layer.

[0140] After forward processing, an output value, called a predicted value, is obtained. To calculate the error, the predicted value is compared with the actual output value to obtain the corresponding error value. Backpropagation uses the chain rule from differential calculus. In this chain rule, the derivatives of the error value with respect to the weights of the last layer of the neural network are first calculated. These derivatives are called gradients. These gradients are then used to calculate the gradients of the penultimate layer of the neural network. This process is repeated until the gradients for each weight in the neural network are obtained. Finally, the corresponding gradient is subtracted from each weight in the neural network to update the weight, thereby reducing the error value.

[0141] For neural networks, fine-tuning involves loading a trained neural network. The fine-tuning process is identical to the training process and consists of two phases: the first is forward signal processing, and the second is backpropagation of gradients to update the weights of the trained neural network. The difference between training and fine-tuning is that training randomly processes an initialized neural network, training the network from scratch, while fine-tuning does not.

[0142] During the training or fine-tuning of a neural network, each time the network undergoes forward signal processing and the corresponding backpropagation of errors, the weights in the network are updated using the gradient. This process is called an iteration. To achieve a neural network with the desired accuracy, the training process requires a very large sample dataset. In this case, it is impossible to input the sample dataset into the computer all at once. Therefore, to solve this problem, the sample dataset is divided into multiple blocks and passed to the computer. After each block is forward processed, the neural network weights are updated. When a complete sample dataset passes through the neural network once and returns a corresponding weight update, this process is called an epoch. In practice, passing the complete dataset through the neural network once is not sufficient; it needs to be passed through the same network multiple times, requiring multiple epochs, to ultimately achieve a neural network with the desired accuracy.

[0143] When training or fine-tuning a neural network, the goal is generally to achieve the fastest possible speed and the highest possible accuracy. Neural network data is represented in high-precision formats, such as floating-point numbers. Therefore, during training or fine-tuning, all data is in high-precision formats. The trained neural network is then quantized. For example, consider the weights of the entire neural network, where the quantized weights are all 8-bit fixed-point numbers. Since a neural network often has millions of connections, almost all the space is occupied by the weights of these connections. Furthermore, these weights are all different floating-point numbers. The weights of each layer tend to be normally distributed within a certain interval, such as (-3.0, 3.0). The maximum and minimum values of the weights for each layer in the neural network are stored, and each floating-point value is represented as an 8-bit fixed-point number. Within the range of the maximum and minimum values, the interval is linearly divided into 256 quantization intervals, each represented by an 8-bit fixed-point number. For example, in the interval (-3.0, 3.0), byte 0 represents -3.0, and byte 255 represents 3.0. Similarly, byte 128 represents 0.

[0144] For data represented in high-precision formats, such as floating-point numbers, computer architecture shows that based on the arithmetic representation rules of floating-point and fixed-point numbers, for fixed-point and floating-point operations of the same length, floating-point calculations are more complex and require more logic devices to construct the floating-point unit. Therefore, floating-point units are physically larger than fixed-point units. Furthermore, floating-point units require more processing resources, resulting in a power consumption difference of orders of magnitude between fixed-point and floating-point operations. In short, floating-point units occupy many times more chip area and consume significantly more power than fixed-point units.

[0145] However, floating-point arithmetic has its own irreplaceable characteristics. First, although fixed-point arithmetic is intuitive, the fixed position of the decimal point determines the integer and decimal parts of a fixed number of digits, which is not conducive to expressing extremely large or extremely small numbers at the same time, and may cause overflow.

[0146] Furthermore, when using AI processor chips for training or fine-tuning, floating-point arithmetic is often preferred. This is primarily because, in supervised learning neural networks, only floating-point arithmetic can record and capture the smallest increments during training. Therefore, how to significantly increase the chip's computing power for training without increasing the chip's area and power consumption is an urgent challenge.

[0147] For those skilled in the art, practical feedback indicates that using fixed-point numbers with low bit widths for training requires processing back-propagation gradients with fixed-point numbers higher than 8 bits, making the training process extremely complex. This specification addresses the technical challenge of replacing floating-point units with fixed-point units to achieve the speed of fixed-point operations, thereby increasing the peak computing power of AI processor chips while maintaining the required floating-point precision.

[0148] Based on the technical issues described above, a characteristic of neural networks is their high tolerance for input noise. For example, considering identifying objects in photos, neural networks can ignore major noise and focus on important similarities. This capability means that neural networks can treat low-precision calculations as a source of noise, still producing accurate predictions even with numerical formats that hold less information. To effectively train or fine-tune low-precision data, it is necessary to find a universally applicable data representation that both mitigates data overflow and better represents data near zero within the target range. Therefore, this data representation needs to be adaptive and adaptable as training or fine-tuning progresses.

[0149] Based on the above description, if Figure 2 As shown in FIG, it is a flow chart of a method for determining quantization parameters of a neural network proposed in this disclosure. Figure 2 The quantization parameters determined by the technical solution shown are used to represent the data to be quantized, thereby determining the fixed-point number after quantization. The quantized fixed-point number is used for training, fine-tuning or inference of a neural network. The method includes:

[0150] Step 201): Count the data to be quantified and determine the statistical results of each type of data to be quantified; wherein the data to be quantified includes at least one of the neurons, weights, gradients, and biases of the neural network.

[0151] As mentioned above, in the process of training or fine-tuning a neural network, each layer of the neural network includes four types of data, namely neurons, weights, gradients, and biases. During the inference process, each layer of the neural network includes three types of data, namely neurons, weights, and biases. These data are all represented in high-precision data format, and this specification takes floating-point numbers as an example of high-precision data. It should be clear that taking floating-point numbers as an example is only an example of some cases, not an exhaustive list. Those skilled in the art, while understanding the essence of this technical solution, may produce other deformations or transformations based on the technical solution of this disclosure, such as: high-precision data can be a fixed-point number with a large representation range and a small minimum precision, and can be converted into a fixed-point number with a low data bit width using this technical solution. However, as long as the functions implemented and the technical effects achieved are similar to those of this disclosure, they should all fall within the scope of protection of this disclosure.

[0152] Regardless of the neural network structure, during the training or fine-tuning of the neural network, the data to be quantized includes at least one of the following: neurons, weights, gradients, and biases. During inference, the data to be quantized includes at least one of the following: neurons, weights, and biases. Taking the example of weights as the data to be quantized, the data to be quantized can be all weights of a layer in the neural network, or part of the weights of a layer in the neural network. If the layer is a convolutional layer, the data to be quantized can also be all weights or part of the weights in the convolutional layer in units of channels, where the channel is all or part of the channels of the convolutional layer. It should be emphasized that only convolutional layers have the concept of channels, and within the convolutional layer, only the weights are quantized in a channel-by-channel manner.

[0153] The following takes the case where the data to be quantified are the neurons and weights of the target layer in the neural network as an example, and describes the technical solution in detail. In this step, the neurons and weights of each layer in the target layer are statistically analyzed to obtain the maximum and minimum values of each type of data to be quantified, and the absolute maximum value of each type of data to be quantified can also be obtained. Among them, the target layer is the layer that needs to be quantified in the neural network, and it can be one layer or multiple layers. Taking one layer as a unit, the absolute maximum value of each type of data to be quantified can be confirmed by the maximum and minimum values of each type of data to be quantified. It is also possible to first calculate the absolute value of each type of data to be quantified, and then traverse the results after calculating the absolute value to obtain the absolute maximum value of each type of data to be quantified.

[0154] In practical applications, the reason for obtaining the absolute maximum value of each data to be quantized according to the maximum and minimum values in each data to be quantized is that, during quantization, the maximum and minimum values corresponding to the data to be quantized in each layer of the target layer are generally saved. There is no need to consume more resources to calculate the absolute value of the data to be quantized, and the absolute maximum value can be directly obtained based on the maximum and minimum values corresponding to the saved data to be quantized.

[0155] Step 202): Determine corresponding quantization parameters using the statistical results and data bit width of each type of data to be quantized; wherein the quantization parameters are used by the artificial intelligence processor to perform corresponding quantization on the data during the neural network operation process.

[0156] In this step, the quantization parameter can be divided into the following six cases. The first case: the quantization parameter is the point position parameter s. In this case, the following formula (1) can be used to quantize the data to be quantized to obtain the quantized data I x :

[0157]

[0158] Among them, s is the point location parameter, I x is the n-bit binary representation of the quantized data x, F x is the floating point value before quantization of data x, and round is the rounding operation. It should be noted that this is not limited to the rounding operation, and other rounding operations can also be used, such as rounding up, rounding down, rounding to zero, etc., to replace the rounding operation in formula (1). At this time, the maximum value A of the floating point number that can be represented by an n-bit fixed-point number is 2 s (2 n-1 -1), then the maximum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is 2 s (2 n-1 -1), the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -2 s(2 n-1 -1). From formula (1), we can see that when the quantization parameter corresponding to the first case is used to quantize the data to be quantized, the quantization interval is 2 s , the quantization interval is recorded as C.

[0159] Let Z be the absolute maximum value of all floating-point numbers in the number field of the data to be quantized, then A needs to contain Z, and Z must be greater than Therefore, the following formula (2) is constrained:

[0160] 2 s (2 n-1 -1)≥Z>2 s-1 (2 n-1 -1) (2)

[0161] therefore, get

[0162] According to formula (3), the n-bit binary representation value I after quantization of data x x Perform inverse quantization to obtain inverse quantized data Wherein, the dequantized data The data format and the corresponding data before quantization F x The data formats are the same and are all floating point values.

[0163]

[0164] The second case: the quantization parameter is the first scaling factor f1. In this case, the quantized data can be quantized using the following formula (4) to obtain the quantized data I x :

[0165]

[0166] Where f1 is the first scaling factor, I x is the n-bit binary representation of the quantized data x, F x is the floating-point value of the data x before quantization, and round is the rounding operation. It should be noted that this is not limited to the rounding operation. Other rounding operations can also be used, such as rounding up, rounding down, and rounding to zero, to replace the rounding operation in formula (4). From formula (4), it can be seen that when the quantization parameter corresponding to the second case is used to quantize the data to be quantized, the quantization interval is f1, and the quantization interval is recorded as C.

[0167] For the first scaling factor f1, there is a case where the point position parameter s is a fixed known value and no longer changes. Let 2 s=T, T is a fixed value, then the maximum value A of a floating point number that can be represented by an n-bit fixed-point number is (2 n-1 -1)×T. In this case, the maximum value A depends on the data bit width n. Let Z be the maximum absolute value of all numbers in the number field of the data to be quantized, then At this time Z=(2 n-1 -1)×f1. The maximum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is (2 n -1 -1)×f1, the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -(2 n-1 -1)×f1. In another case, in engineering applications, 2 s ×f2 as a whole is regarded as the first scaling factor f1. At this time, it can be regarded as if there is no independent point position parameter s. Among them, f2 is the second scaling factor. Let Z be the maximum absolute value of all numbers in the number field of the data to be quantized, then At this time Z=(2 n-1 -1)×f1. The maximum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is (2 n-1 -1)×f1, the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -(2 n-1 -1)×f1.

[0168] According to formula (5), the n-bit binary representation value I after quantization of data x x Perform dequantization to obtain dequantized data Wherein, the dequantized data The data format and the corresponding data before quantization F x The data formats are the same and are all floating point values.

[0169]

[0170] The third case: the quantization parameters are the point position parameter s and the second scaling factor f2. In this case, the quantization data can be quantized using the following formula (6) to obtain the quantized data I x :

[0171]

[0172] Among them, s is the point position parameter, f2 is the second scaling factor, I x is the n-bit binary representation of the quantized data x, F xis the floating point value of the data x before quantization, and round is the rounding operation. It should be noted that this is not limited to the rounding operation, and other rounding operations can also be used, for example, rounding up, rounding down, rounding to zero, etc., to replace the rounding operation in formula (6). The maximum value A in the number domain of the data to be quantized that can be represented by an n-bit fixed-point number is 2 s (2 n-1 -1). From formula (6), we can see that when the quantization parameters corresponding to the third case are used to quantize the data to be quantized, the quantization interval is 2 s ×f2, the quantization interval is denoted as C.

[0173] Let Z be the maximum absolute value of all numbers in the number domain of the data to be quantized. At this time, according to formula (2), we can get:

[0174] Right now

[0175] When f2=1, formula (6) and formula (1) are the same. The maximum value in the domain of the data to be quantized that can be represented by an n-bit fixed-point number is (2 n-1 -1)×2 s ×f2, the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -(2 n-1 -1)×2 s ×f2.

[0176] According to formula (7), the n-bit binary representation value I after quantization of data x x Perform dequantization to obtain dequantized data Wherein, the dequantized data The data format and the corresponding data before quantization F x The data formats are the same and are all floating point values.

[0177]

[0178] like Figure 3 As shown, a schematic diagram of symmetrical fixed-point number representation is shown. Figure 3 The number domain of the data to be quantized is symmetrically distributed with "0" as the center. Z is the maximum absolute value of all floating-point numbers in the number domain of the data to be quantized. Figure 3 In the example, A is the maximum floating point number that can be represented by an n-bit fixed-point number. The floating point number A converted to a fixed-point number is 2 n-1-1. To avoid overflow, A needs to include Z. In practice, floating-point data in neural network operations tends to be normally distributed within a certain interval, but does not necessarily satisfy the distribution centered on "0". In this case, overflow is likely to occur when represented by fixed-point numbers. To improve this situation, an offset is introduced into the quantization parameter, such as Figure 4 As shown. Figure 4 In the quantized data, the number domain is not symmetrically distributed with "0" as the center, Z min is the minimum value of all floating-point numbers in the number domain of the data to be quantized, Z max It is the maximum value of all floating point numbers in the number domain of the data to be quantized. min ~Z max The center point between the two, the number domain of the data to be quantized is shifted as a whole, so that the number domain of the data to be quantized after the translation is distributed symmetrically with "0" as the center, and the maximum absolute value in the number domain of the data to be quantized after the translation is Z. Figure 4 It can be seen that the offset is the horizontal distance between point "0" and point "P", which is called offset O.

[0179] Based on the above description of the offset O, a fourth quantization parameter situation occurs. The fourth situation: the quantization parameter includes the point position parameter and the offset. In this case, the following formula (8) can be used to quantize the data to be quantized to obtain the quantized data I x :

[0180]

[0181] Among them, s is the point position parameter, O is the offset, I x is the n-bit binary representation of the quantized data x, F x is the floating point value before quantization of data x, and round is the rounding operation. It should be noted that this is not limited to the rounding operation, and other rounding operations can also be used, such as rounding up, rounding down, rounding to zero, etc., to replace the rounding operation in formula (8). At this time, the maximum value A of the floating point number that can be represented by an n-bit fixed-point number is 2 s (2 n-1 -1), then the maximum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is 2 s (2 n-1 -1)+O, the minimum value in the number field of the n-bit fixed-point number that can represent the data to be quantized is -2 s (2 n-1 -1)+O. From formula (8), we can see that when the quantization parameter corresponding to the fourth case is used to quantize the data to be quantized, the quantization interval is 2s , the quantization interval is recorded as C.

[0182] Let Z be the maximum absolute value of all floating-point numbers in the number domain of the data to be quantized, Then A needs to contain Z, and Z must be greater than According to formula (2), we can obtain Then get

[0183] According to formula (9), the n-bit binary representation value I after quantization of data x x Perform dequantization to obtain dequantized data Wherein, the dequantized data The data format and the corresponding data before quantization F x The data formats are the same and are all floating point values.

[0184]

[0185] Based on the above description of the offset O, a fifth quantization parameter situation occurs. The fifth situation: the quantization parameter includes the first scaling factor f1 and the offset O. In this case, the following formula (10) can be used to quantize the data to be quantized to obtain the quantized data I x :

[0186]

[0187] Where f1 is the first scaling factor, O is the offset, and I x is the n-bit binary representation of the quantized data x, F x is the floating point value before quantization of data x, and round is the rounding operation. It should be noted that this is not limited to the rounding operation. Other rounding operations can also be used, such as rounding up, rounding down, rounding to zero, etc., to replace the rounding operation in formula (10). At this time, there is a case where the point position parameter s is a fixed known value and no longer changes. Let 2 s =T, T is a fixed value. Then, the maximum value A of a floating point number that can be represented by an n-bit fixed-point number is (2 n-1 -1)×T. In this case, the maximum value A depends on the data bit width n. Let Z be the maximum absolute value of all numbers in the number field of the data to be quantized, then At this time Z=(2 n-1 -1)×f1. The maximum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is (2 n-1 -1)×f1, the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -(2 n-1-1)×f1. In another case, in engineering applications, 2 s ×f2 as a whole is regarded as the first scaling factor f1. At this time, it can be regarded as if there is no independent point position parameter s. Among them, f2 is the second scaling factor. Let Z be the maximum absolute value of all numbers in the number field of the data to be quantized, then At this time Z=(2 n-1 -1)×f1. The maximum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is (2 n-1 -1)×f1+O, the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -(2 n-1 -1)×f1+O.

[0188] It can be seen from formula (10) that when the quantization parameter corresponding to the fifth case is used to quantize the data to be quantized, the quantization interval is f1, and the quantization interval is recorded as C.

[0189] According to formula (11), the n-bit binary representation value I after quantization of data x x Perform dequantization to obtain dequantized data Wherein, the dequantized data The data format and the corresponding data before quantization F x The data formats are the same and are all floating point values.

[0190]

[0191] Based on the above description of the offset O, a sixth quantization parameter situation appears. The sixth situation: the quantization parameters include the point position parameter, the second scaling factor f2 and the offset O. In this case, the following formula (12) can be used to quantize the data to be quantized to obtain the quantized data I x :

[0192]

[0193] Among them, s is the point position parameter, the offset is O, and f2 is the second scaling factor. I x is the n-bit binary representation of the quantized data x, F x is the floating point value of data x before quantization, and round is the rounding operation. It should be noted that this is not limited to the rounding operation, and other rounding operations can also be used, such as rounding up, rounding down, rounding to zero, etc., to replace the rounding operation in formula (12). The maximum value A in the number domain of the data to be quantized that can be represented by an n-bit fixed-point number is 2 s (2 n-1-1). From formula (12), we can see that when the quantization parameter corresponding to the sixth case is used to quantize the data to be quantized, the quantization interval is 2 s ×f2, the quantization interval is denoted as C.

[0194] Let Z be the maximum absolute value of all numbers in the number domain of the data to be quantized. At this time, according to formula (2), we can get:

[0195] Right now

[0196] When , according to formula (2), Z can be accurately expressed without loss. When f2=1, The maximum value in the domain of the data to be quantized that can be represented by an n-bit fixed-point number is (2 n-1 -1)×2 s ×f2+O, the minimum value in the number field of the data to be quantized that can be represented by an n-bit fixed-point number is -(2 n-1 -1)×2 s ×f2+O.

[0197] According to formula (13), the n-bit binary representation value I after quantization of data x x Perform inverse quantization to obtain inverse quantized data Wherein, the dequantized data The data format and the corresponding data before quantization F x The data formats are the same and are all floating point values.

[0198]

[0199] The above detailed description of the determination process of the six quantization parameters is merely an example. The types of quantization parameters may differ from the above description in different embodiments. As can be seen from formulas (1) to (13), the point position parameter and the scaling factor are both related to the data bit width. Different data bit widths result in different point position parameters and scaling factors, which in turn affect the quantization accuracy. During training or fine-tuning, within a certain range of iterations, using the same data bit width for quantization has little effect on the overall accuracy of the neural network operation. Beyond a certain number of iterations, using the same data bit width for quantization will not meet the accuracy requirements of training or fine-tuning. This requires adjusting the data bit width n as the training or fine-tuning process progresses. Simply put, the data bit width n can be manually set. Within different ranges of iterations, the corresponding data bit width n set in advance is called. However, as mentioned above, the process of implementing training using fixed-point numbers represented by low bit widths is extremely complex. This method of manually setting the data bit width in advance is basically not in line with the needs of practical applications.

[0200] In this technical solution, according to the quantization error diff bit Adjust the data bit width n. To further elaborate, the quantization error diff bit Compare with the threshold and obtain the comparison result. Among them, the threshold includes a first threshold and a second threshold, and the first threshold is greater than the second threshold. There are three cases for the comparison result. The first case is: quantization error diff bit is greater than or equal to the first threshold, in this case, the data bit width is increased. The second case is: the quantization error diff bit is less than or equal to the second threshold, in this case, the data bit width is reduced. The third case is: the quantization error diff bit The first threshold value and the second threshold value are between the first threshold value and the second threshold value. In this case, the data bit width remains unchanged. In practical applications, the first threshold value and the second threshold value can be empirical values or variable hyperparameters. Conventional hyperparameter optimization methods are applicable to the first threshold value and the second threshold value, and the hyperparameter optimization scheme is not further described here.

[0201] It should be emphasized that the data bit width can be adjusted according to a fixed bit step size, or according to the difference between the quantization error and the error threshold, the data bit width can be adjusted according to a variable adjustment step size. Finally, the data bit width can be adjusted longer or shorter according to the actual needs of the neural network operation process. For example: the data bit width n of the current convolution layer is 16, according to the quantization error diff bit Adjust the data bit width n to 12. In other words, in practical applications, the data bit width n can be set to 12 instead of 16 to meet the accuracy requirements of the neural network operation process. This can greatly increase the fixed-point operation speed within the accuracy range, thereby improving the resource utilization of the artificial intelligence processor chip.

[0202] For the quantization error diff bit For example, the quantization error is determined based on the quantized data and the corresponding pre-quantization data. In practical applications, there are three methods for determining the quantization error, all of which are applicable to this technical solution. The first method is to determine the quantization error according to formula (14) based on the quantization interval, the number of quantized data, and the corresponding pre-quantization data.

[0203]

[0204] Among them, C is the quantization interval corresponding to quantization, m is the number of quantized data obtained after quantization, and F i is the floating-point value corresponding to the quantization, where i is the subscript of the data in the data set to be quantized.

[0205] The second method is to determine the quantization error diff according to the quantized data and the corresponding dequantized data according to formula (15) bit .

[0206]

[0207] Among them, F i is the floating-point value corresponding to the quantization, where i is the subscript of the data in the data set to be quantized. The inverse quantized data corresponding to the floating-point value.

[0208] The third method: Determine the quantization error diff according to formula (16) based on the quantized data and the corresponding dequantized data bit .

[0209]

[0210] Among them, F i is the floating-point value corresponding to the quantization, where i is the subscript of the data in the data set to be quantized. The inverse quantized data corresponding to the floating-point value.

[0211] It should be emphasized that the above quantization error diff bit The above-mentioned methods are merely examples of some cases, not exhaustive. Those skilled in the art, while understanding the essence of the technical solution disclosed herein, may produce other deformations or transformations based on the technical solution disclosed herein. Any deformation formula that supports determining the quantization error based on the quantized data and the corresponding pre-quantization data, as long as the functions it implements and the technical effects it achieves are similar to those disclosed herein, shall fall within the scope of protection of the present disclosure.

[0212] Regarding data bit width, Figure 5a shows the first graph of the weight data fluctuation amplitude during training. Figure 5b shows the second graph of the weight data fluctuation amplitude during training. In Figures 5a and 5b, the horizontal axis represents the iteration number, and the vertical axis represents the maximum value after taking the logarithm of the weight. The weight data fluctuation amplitude curve shown in Figure 5a shows the weight data fluctuation corresponding to different iterations within the same epoch for any convolutional layer of the neural network. In Figure 5b, the weight data fluctuation amplitude curve A corresponds to the conv0 layer, the weight data fluctuation amplitude curve B corresponds to the conv1 layer, the weight data fluctuation amplitude curve C corresponds to the conv2 layer, the weight data fluctuation amplitude curve D corresponds to the conv3 layer, and the weight data fluctuation amplitude curve e corresponds to the conv4 layer. As shown in Figures 5a and 5b, within the same epoch, in the early stages of training, the weight fluctuation amplitude with each iteration is relatively large. In the middle and late stages of training, the weight fluctuation amplitude with each iteration is not much larger. In this case, in the middle and late stages of training, because the weight data does not change much before and after each iteration, the weight data of the corresponding layers of each generation are similar within a certain iteration interval. During the neural network training process, the data quantization of each layer involved can use the data bit width used when quantizing the corresponding layer in the previous iteration. However, in the early stages of training, since the weight data changes greatly before and after each iteration, in order to meet the floating-point operation accuracy required for quantization, in each iteration of the early training, the weight data of the corresponding layer of the current generation is quantized using the data bit width used when quantizing the corresponding layer in the previous iteration, or the weight data of the current layer is quantized based on the preset data bit width n of the current layer to obtain the quantized fixed-point number. Based on the quantized weight data and the corresponding weight data before quantization, the quantization error diff is determined. bit , according to the quantization error diff bit Based on the comparison result with the threshold, the data bit width n used in the quantization of the corresponding layer in the previous iteration or the preset data bit width n of the current layer is adjusted, and the adjusted data bit width is applied to the quantization of the weight data of the corresponding layer in the current iteration. Furthermore, during the training or fine-tuning process, the weight data between each layer of the neural network are independent of each other and lack similarity. Because the weight data lacks similarity, the neuron data between each layer is also independent of each other and lacks similarity. Therefore, during the training or fine-tuning process of the neural network, the data bit width of each layer in each iteration of the neural network is only applicable to the corresponding neural network layer.

[0213] The above takes weight data as an example. During the neural network training or fine-tuning process, the data bit widths corresponding to neuron data and gradient data are also the same, which will not be repeated here.

[0214] During the neural network inference process, the weight data between each layer of the neural network are independent of each other and have no similarity. Because the weight data have no similarity, the neuron data between each layer are also independent of each other and have no similarity. Therefore, during the neural network inference process, the data bit width of each layer of the neural network is applied to the corresponding layer. In actual applications, the input neuron data of each time during the inference process is likely to be different or dissimilar, and because the weight data between each layer of the neural network are independent of each other, the input neuron data of each layer in the hidden layer of the neural network are not similar. During quantization, the data bit width used for the input neuron data of the previous layer is not suitable for the input neuron data of the current layer. Based on this, in order to meet the floating-point operation accuracy required for quantization, during inference, the input neuron data of the current layer is quantized using the data bit width used when quantizing the input neuron data of the previous layer, or the input neuron data of the current layer is quantized based on the preset data bit width n of the current layer to obtain the quantized fixed-point number. The quantization error diff is determined based on the input neuron data before quantization and the corresponding input neuron data after quantization. bit , according to the quantization error diff bit The comparison result with the threshold is used to adjust the data bit width n used when quantizing the input neuron data of the previous layer, or the preset data bit width n of the current layer. The adjusted data bit width is then applied to the quantization of the input neuron data of the current layer. The data bit width corresponding to the weight data is also processed in the same way and will not be further described here.

[0215] As for the quantization parameters, Figure 5a shows that within the same epoch, in the early stages of training, the weights change significantly with each iteration. In the middle and late stages of training, because the weight data changes little before and after each iteration, the weight data of the corresponding layers in each iteration remain similar within a certain interval. Therefore, during quantization, the data of each layer in the current iteration can use the quantization parameters of the corresponding layer in the previous iteration. In the middle and late stages of training, the quantization parameters do not need to be reconfirmed for each generation. They are only confirmed for each layer in the early stages of training. This still meets the floating-point accuracy required for neural network operations, greatly improving quantization efficiency. Furthermore, during training or fine-tuning, the weight data of each layer in the neural network is independent and lacks similarity. This lack of similarity in weight data also leads to independence and lack of similarity in the neuron data of each layer. Therefore, during training or fine-tuning, the quantization parameters of each layer in each iteration of the neural network are applied to the corresponding data to be quantized in the corresponding layer.

[0216] The above takes weight data as an example. During the neural network training or fine-tuning process, the quantization parameters corresponding to the neuron data and gradient data are also the same, which will not be repeated here.

[0217] During the neural network inference process, the weight data between each layer of the neural network are independent of each other and have no similarity. Because the weight data have no similarity, the neuron data between each layer are also independent of each other and have no similarity. Therefore, during the neural network inference process, the quantization parameters of each layer of the neural network are applied to the data to be quantized in the corresponding layer. For example: the current layer of the neural network is the convolution layer, according to the data to be quantized in the convolution layer, Figure 2 The technical solution shown obtains the quantization parameter of the data to be quantized in the current convolution layer. The quantization parameter can only be applied to the current convolution layer and cannot be applied to other layers of the neural network, even if the other layers are convolution layers.

[0218] In summary, the data width and quantization parameter extension strategy is determined based on the similarity between the data. If the data are similar, the data width and quantization parameters can be extended. If the data are not similar, the data width or quantization parameters need to be adjusted. The similarity between data is usually measured using the KL divergence, but can also be measured using the following formula (17).

[0219] abs max(A)≈abs max(B) and mean(A)≈mean(B) (17)

[0220] In some embodiments, if data A and data B satisfy formula (17), it is determined that data A and data B have similarity.

[0221] It should be noted that the above-described methods for confirming quantization errors, adjusting data bit widths, and strategies for extending data bit widths and quantization parameters are merely examples of some cases, not exhaustive. For example, the above-described methods for confirming quantization errors, adjusting data bit widths, and strategies for extending data bit widths and quantization parameters are all applicable to the fine-tuning process of neural networks. In addition, regarding the measurement of similarity between data, the above-mentioned KL divergence and the similarity measurement method of formula (17) are merely examples of some cases, not exhaustive, such as histogram matching method, matrix decomposition method, feature point-based image similarity calculation method, proximity measurement standard method, etc. Those skilled in the art may produce other deformations or transformations based on the technical solution of this disclosure while understanding the essence of the technical solution of this disclosure. However, as long as the functions implemented and the technical effects achieved are similar to those of this disclosure, they should all fall within the scope of protection of this disclosure.

[0222] In summary, in the middle and late stages of training, because the weight data does not change much before and after each iteration, the weight data of the corresponding layers of each iteration are similar within a certain iteration interval. In order to make this technical solution more universal in training or fine-tuning and to meet the reasonable application of the resources of the artificial intelligence processor chip, a strategy is needed to determine the iteration interval so that within the iteration interval, the data bit width n of the corresponding layer of each iteration remains unchanged. If the iteration interval is exceeded, the data bit width n will change, and there is no need to determine whether to adjust the data bit width n from generation to generation. Similarly, the quantization parameter is also the same, so as to achieve the goal of improving the peak computing power of the artificial intelligence processor chip while meeting the accuracy of the floating-point operations required for quantization.

[0223] like Figure 6 As shown in FIG, it is one of the flow charts of the method for determining the target iteration interval. Figure 6 In the technical solution shown, the target iteration interval includes at least one weight update iteration, and the same data bit width is used in the quantization process within the same target iteration interval. The step of determining the target iteration interval includes:

[0224] Step 601): At a predicted time point, determine the change trend value of the position parameter of the corresponding point of the data to be quantized during the weight iteration process; wherein, the predicted time point is a time point for judging whether the data bit width needs to be adjusted, and the predicted time point corresponds to the time point when the weight update iteration is completed.

[0225] In this step, according to formula (18), the change trend value of the point position parameter is determined based on the sliding average of the point position parameter in the weight iteration process corresponding to the current prediction time point and the sliding average of the point position parameter in the weight iteration process corresponding to the previous prediction time point, or based on the sliding average of the point position parameter in the weight iteration process corresponding to the current prediction time point and the sliding average of the point position parameter in the weight iteration process corresponding to the previous prediction time point. Formula 18 is expressed as:

[0226] diff update1 =|M (t) -M (t-1) |=α|s (t) -M (t-1) | (18)

[0227] In formula 18, M is the sliding average of the point position parameter s as the training iteration increases. (t) is the sliding average value of the point position parameter s corresponding to the t-th prediction time point as the training iteration increases. According to formula (19), M is obtained (t) . s (t) M is the point position parameter s corresponding to the t-th predicted time point. (t-1)is the sliding average of the point position parameter s corresponding to the t-1th predicted time point, and α is a hyperparameter. update1 Measure the trend of the point location parameter s. Since the change of the point location parameter s is also reflected in the maximum value Z of the data to be quantified max Changes in diff update1 The larger it is, the more dramatic the change in the value range is, and a shorter update frequency is required, that is, the target iteration interval is smaller.

[0228] M (t) ←α×s (t-1) +(1-α)×M (t-1) (19)

[0229] Step 602): Determine the target iteration interval corresponding to the target iteration interval according to the change trend value of the point position parameter.

[0230] In this technical solution, the target iteration interval is determined according to formula (20). For the target iteration interval, the same data bit width is used in the quantization process within the same target iteration interval, and the data bit widths used in the quantization process within different target iteration intervals can be the same or different.

[0231]

[0232] In formula (20), I is the target iteration interval. update1 is the trend value of the point location parameter. β and γ are empirical values and can also be variable hyperparameters. Conventional hyperparameter optimization methods are applicable to β and γ, so we will not elaborate on the hyperparameter optimization scheme here.

[0233] For this technical solution, the predicted time point includes a first predicted time point, which is determined according to the target iteration interval. Specifically, at the t-th predicted time point during the training or fine-tuning process, the weight data of the corresponding layer of the current iteration is quantized using the data bit width used when quantizing the corresponding layer of the previous iteration to obtain the quantized fixed-point number, and the quantization error diff is determined based on the weight data before quantization and the corresponding weight data before quantization. bit . The quantization error diff bit Compare with the first threshold and the second threshold respectively, and use the comparison results to determine whether to adjust the data bit width used in the quantization of the corresponding layer in the previous iteration. Suppose: the t-th first prediction time point corresponds to the 100th iteration, and the data bit width used in the 99th iteration is n1. At the 100th iteration, the quantization error diff is confirmed based on the data bit width n1 bit , the quantization error diff bitCompare with the first threshold and the second threshold to obtain a comparison result. If the data bit width n1 does not need to be changed according to the comparison result, use formula (20) to confirm that the target iteration interval is 8 iterations. When the 100th iteration is used as the starting iteration in the current target iteration interval, then the 100th to 107th iterations are used as the current target iteration interval. When the 100th iteration is used as the last iteration in the previous target iteration interval, then the 101st to 108th iterations are used as the current target iteration interval. When quantizing in the current target iteration interval, each generation still uses the data bit width n1 used in the previous target iteration interval. In this case, the data bit width used for quantization between different target iteration intervals can be the same. If the 100th to 107th iterations are used as the current target iteration interval, then the 108th iteration in the next target iteration interval is used as the t+1th first prediction time point. If the 101st to 108th iterations are used as the current target iteration interval, then the 108th iteration in the current target iteration interval is used as the t+1th first prediction time point. At the first predicted time point t+1, the quantization error diff is confirmed according to the data bit width n1. bit , the quantization error diff bit Compare with the first threshold and the second threshold to obtain a comparison result. Based on the comparison result, determine that the data bit width n1 needs to be changed to n2, and use formula (20) to confirm that the target iteration interval is 55 iterations. Then, the 108th to 163rd iteration or the 109th to 163rd iteration is used as the target iteration interval, and the data bit width n2 is used for each generation during quantization within this target iteration interval. In this case, the data bit width used for quantization can be different between different target iteration intervals.

[0234] For this technical solution, regardless of whether the first predicted time point is the starting iteration or the last iteration within the target iteration interval, formula (18) is applicable to obtain the change trend value of the point position parameter. If the first predicted time point at the current moment is the starting iteration of the current target iteration interval, then in formula (18), M (t) is the sliding average value of the point position parameter s corresponding to the time point corresponding to the starting iteration of the current target iteration interval as the training iteration increases, s (t) The point position parameter s corresponding to the time point corresponding to the starting iteration of the current target iteration interval, M (t-1) is the sliding average of the point position parameter s corresponding to the time point corresponding to the starting iteration of the previous target iteration interval as the training iteration increases. If the first predicted time point at the current moment is the last iteration of the current target iteration interval, then in formula (18), M (t) is the sliding average value of the point position parameter s corresponding to the time point corresponding to the last iteration of the current target iteration interval as the training iteration increases, s(t) is the point position parameter s corresponding to the time point corresponding to the last iteration of the current target iteration interval, M (t-1) It is the sliding average of the point position parameter s corresponding to the time point corresponding to the last iteration of the previous target iteration interval as the training iteration increases.

[0235] In this technical solution, in addition to the first predicted time point, the predicted time point may also include a second predicted time point. The second predicted time point is determined based on the data variation amplitude curve. Based on the data variation amplitude of the big data during the neural network training process, the data variation amplitude curve shown in Figure 5a is obtained.

[0236] Taking the weight data as an example, the data variation curve shown in Figure 5a shows that from the start of training to the Tth iteration, the data variation is very large each time the weight is updated. At the current prediction time point, during quantization, the current iteration first uses the data bit width n1 of the previous iteration for quantization. The obtained quantization result is compared with the corresponding pre-quantization data to determine the corresponding quantization error. The quantization error is compared with the first threshold and the second threshold respectively. According to the comparison result, the data bit width n1 is adjusted to obtain the data bit width n2. The weight data to be quantized in the current iteration is quantized using the data bit width n2. Then, the target iteration interval is determined according to formula (20), thereby determining the first prediction time point. At the first prediction time point, it is determined whether to adjust the data bit width and how to adjust it. The next target iteration interval is determined according to formula (20) to obtain the next first prediction time point. Since the weight data changes greatly before and after each iteration during the iteration interval from the start of training to the Tth iteration, the weight data of the corresponding layers of each iteration are not similar. In order to meet the accuracy problem, the data of each layer of the current iteration cannot be quantized using the corresponding quantization parameters of the corresponding layer of the previous iteration. The data bit width can be adjusted from generation to generation in the first T iterations. At this time, the data bit width used in each iteration of the first T iterations is different, and the target iteration interval is 1 iteration. In order to achieve optimal utilization of the resources of the artificial intelligence processor chip, the target iteration interval of the first T iterations can be preset in advance according to the law revealed by the data variation amplitude curve shown in Figure 5a, that is, the target iteration interval of the first T iterations can be directly preset according to the data variation amplitude curve, without the need to confirm the time point when the weight update iteration corresponding to each iteration of the first T iterations is completed as the second predicted time point through formula (20). This makes the resources of the artificial intelligence processor chip more reasonably utilized. The data variation curve shown in Figure 5a shows a small variation starting from the Tth iteration. In the middle and late stages of training, the quantization parameters do not need to be reconfirmed for each generation. At the Tth iteration or the T+1th iteration, the quantization error is determined using the data before and after quantization corresponding to the current iteration. Based on the quantization error, it is determined whether the data bit width needs to be adjusted and how to adjust it. The target iteration interval is also determined according to formula (20). If the confirmed target iteration interval is 55 iterations, this requires that the time point corresponding to 55 iterations after the Tth iteration or the T+1th iteration be used as the first prediction time point to determine whether the data bit width needs to be adjusted and how to adjust it. The next target iteration interval is determined according to formula (20), thereby determining the next first prediction time point until all generations within the same cycle (epoch) are completed. On this basis, after each cycle (epoch), the data bit width or quantization parameter is adaptively adjusted, and finally the quantized data is used to obtain a neural network with the expected accuracy.

[0237] In particular, if: based on the weight data variation curve shown in Figure 5a, it is determined that T is 130 (this value does not correspond to Figure 5a. For the convenience of description, it is only assumed that T is 130, and it is not limited to the assumed value.), then the 130th iteration in the training process is used as the second prediction time point, and the current first prediction time point is the 100th iteration in the training process. At the 100th iteration, the target iteration interval is determined to be 35 iterations according to formula (20). Within this target iteration interval, training is carried out to the 130th iteration, reaching the second prediction time point. At this time, it is necessary to determine whether the data bit width needs to be adjusted and how to adjust it at the time point corresponding to the 130th iteration, and the target iteration interval is also determined according to formula (20). Suppose the target iteration interval determined in this case is 42 iterations. The target iteration interval is from the 130th iteration to the 172nd iteration. The 135th iteration corresponding to the first prediction time point determined when the target iteration interval is 35 iterations is within the target iteration interval of 42 iterations. At the 135th iteration, it can be determined whether the data bit width needs to be adjusted and how to adjust it according to formula (20). Alternatively, it is possible to skip the evaluation and prediction at the 135th iteration and directly perform the evaluation and adjustment of whether the data bit width needs to be adjusted at the 172nd iteration. In short, whether the evaluation and prediction are performed at the 135th iteration is suitable for this technical solution.

[0238] In summary, the second prediction time point is preset in advance according to the data change amplitude curve. In the early stage of training or fine-tuning, there is no need to spend the resources of the artificial intelligence processor chip to determine the target iteration interval. At the preset second prediction time point, the data bit width is directly adjusted according to the quantization error, and the adjusted data bit width is used to quantize the data to be quantized involved in the current iteration. In the middle and late stages of training or fine-tuning, the target iteration interval is obtained according to formula (20), thereby determining the corresponding first prediction time point, and determining whether to adjust the data bit width and how to adjust it at each first prediction time point. In this way, while meeting the accuracy of floating-point operations required for neural network operations, the resources of the artificial intelligence processor chip are reasonably utilized, greatly improving the efficiency of quantization.

[0239] In practice, in order to obtain a more accurate target iteration interval of data width, it is not only based on the change trend value diff of the point position parameter update1 , you can also consider the change trend value diff of the point location parameter update1 and the change trend value diff of data bit width update2 .like Figure 7 The figure shows the second flow chart of the method for determining the target iteration interval. The steps of determining the target iteration interval include:

[0240] Step 701): At a predicted time point, determine the change trend value of the position parameter of the corresponding point of the data to be quantized and the change trend value of the data bit width during the weight iteration process; wherein, the predicted time point is a time point for judging whether the data bit width needs to be adjusted, and the predicted time point corresponds to the time point when the weight update iteration is completed.

[0241] It should be emphasized that Figure 6 The technical solution for determining the target iteration interval of data bit width based on the change trend value of the point position parameter is applicable to Figure 7 The technical solution shown will not be described in detail here.

[0242] In this step, the change trend value of the data bit width is determined using the corresponding quantization error according to formula (21).

[0243]

[0244] In formula (21), δ is a hyperparameter, diff bit is the quantization error; diff update2 The change trend value of the data width. update2 Measures the changing trend of the data bit width n used in quantization, diff update2 The larger the value, the more likely it is that the fixed-point bit width needs to be updated, and a shorter update frequency is required.

[0245] for Figure 7 The trend value of the point position parameters involved in can still be obtained according to formula (18). For M in formula (18), (t) According to formula (19), diff update1 Measure the trend of the point location parameter s. Since the change of the point location parameter s is also reflected in the maximum value Z of the data to be quantified max Changes in diff update1 The larger it is, the more dramatic the change in the value range is, and a shorter update frequency is required, that is, the target iteration interval is smaller.

[0246] Step 702): Determine the target iteration interval corresponding to the target iteration interval according to the change trend value of the point position parameter and the change trend value of the data bit width.

[0247] In this technical solution, the target iteration interval is determined according to formula (22). For the target iteration interval, the same data bit width is used in the quantization process within the same target iteration interval, and the data bit widths used in the quantization process within different target iteration intervals can be the same or different.

[0248]

[0249] In formula (22), I is the target iteration interval. β and γ are hyper parameters. diff update1 The change trend value of the point position parameter. update2 is the trend value of the data width. β and γ are empirical values and can also be variable hyperparameters. Conventional hyperparameter optimization methods are applicable to β and γ, so we will not elaborate on the hyperparameter optimization scheme here.

[0250] For this technical solution, diff update1 It is used to measure the change of the point position parameter s, but the change of the point position parameter s caused by the change of the data bit width n should be ignored. update2 The change of data bit width n is reflected in the diff update1 If this ignore operation is not performed, the target iteration interval I determined by formula (22) is inaccurate, resulting in too many first prediction time points. During the training or fine-tuning process, it is easy to frequently perform operations on whether to update the data bit width n and how to update it, resulting in unreasonable utilization of the resources of the artificial intelligence processor chip.

[0251] Based on the above description, diff update1 According to M (t) Determine. Assume that the data width corresponding to the t-1th predicted time point is n1, the corresponding point position parameter is s1, and the sliding average of the point position parameter as the training iteration increases is m1. Use the data width n1 to quantize the data to be quantized and obtain the quantized fixed-point number. Determine the quantization error diff based on the data before quantization and the corresponding quantized data. bit , according to the quantization error diff bit Compared with the threshold, the data width n1 is adjusted to n2, and the data width is adjusted by |n1-n2| bits. The data width used for quantization of the t-th predicted time point is n2. In order to ignore the change of the point position parameter caused by the change of the data width, when determining M (t) You can choose one of the following two optimization methods. The first method: If the data bit width increases by |n1-n2| bits, then s (t-1) The value is s1-|n1-n2|, M (t-1) Take the value as m1-|n1-n2|, and change s (t-1) 、M (t-1) Substituting into formula (19), we get M (t) , which is the sliding average of the point position parameter corresponding to the t-th predicted time point as the training iteration increases. If the data bit width is reduced by |n1-n2| bits, then s (t -1) The value is s1+|n1-n2|, M (t-1)Take the value of m1+|n1-n2|, and change s (t-1) 、M (t-1) Substituting into formula (19), we get M (t) , which is the sliding average of the point position parameter corresponding to the t-th predicted time point as the training iteration increases. The second method: Regardless of whether the data bit width increases by |n1-n2| bits or decreases by |n1-n2|, s (t-1) The value is s1, M (t-1) Take the value as m1, and change s (t-1) 、M (t-1) Substituting into formula (19), we get M (t) When the data width increases by |n1-n2| bits, M (t) Subtract |n1-n2|, when the data width is reduced by |n1-n2| bits, M (t) Add |n1-n2|, and the result is the sliding average of the point position parameters corresponding to the tth predicted time point as training iterations increase. These two methods are equivalent and both can ignore changes in point position parameters caused by changes in data bit width, achieving a more accurate target iteration interval, thereby improving resource utilization of the AI processor chip.

[0252] In practical applications, the data bit width n and the point position parameter s have a great influence on the quantization accuracy, while the second scaling factor f2 and the offset O in the quantization parameter have little influence on the quantization accuracy. As mentioned above, for the first scaling factor f1, if it belongs to the second case, 2 s ×f2 as a whole is taken as the first scaling factor f1. Since the point position parameter s has a great influence on the quantization accuracy, the first scaling factor f1 in this case has a great influence on the quantization. Therefore, in this technical solution, regardless of whether the data bit width n changes and the point position parameter s is variable, it is also very meaningful to determine the target iteration interval of the point position parameter s. Figure 6 The idea of the technical solution shown can be applied to determine the target iteration interval of the point position parameter s. Therefore, the method for determining the target iteration interval of the point position parameter s is as follows: Figure 8A As shown. Includes:

[0253] Step 801): At a predicted time point, determine the change trend value of the position parameter of the corresponding point of the data to be quantized involved in the weight iteration process; wherein, the predicted time point is a time point for judging whether the quantization parameter needs to be adjusted, and the predicted time point corresponds to the time point when the weight update iteration is completed.

[0254] Step 802): Determine the target iteration interval corresponding to the target iteration interval according to the change trend value of the point position parameter.

[0255] It needs to be emphasized that Figure 6 The technical solution for determining the target iteration interval of the quantization parameter based on the change trend value of the point position parameter is applicable to Figure 8A The technical solution shown in FIG. Figure 8A For the technical solution shown, the quantization parameter is preferably a point position parameter.

[0256] It should be noted that the above-mentioned target iteration intervals for determining the data bit width and the target iteration intervals for the quantization parameters are only examples of some cases, not exhaustive. Those skilled in the art, while understanding the essence of the disclosed technical solution, may produce other deformations or transformations based on the disclosed technical solution. For example, determining the target iteration interval for the quantization parameters within the target iteration interval for determining the data bit width is also applicable to Figure 6 、 Figure 7 and Figure 8A However, as long as the functions and technical effects achieved are similar to those of this disclosure, they should fall within the scope of protection of this disclosure.

[0257] This technical solution is used to determine the quantization parameter, adjust the data bit width or quantization parameter according to the quantization error, and determine the target iteration interval for adjusting the data bit width or quantization parameter, so as to adjust the data bit width or quantization parameter at an appropriate time point during the neural network operation, so that the appropriate quantization parameter is used at the appropriate iteration time point, and the artificial intelligence processor chip can execute the neural network operation at the speed of fixed-point operation, thereby improving the peak computing power of the artificial intelligence processor chip while meeting the accuracy of floating-point operations required for the operation.

[0258] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this disclosure is not limited by the order of the actions described, because according to this disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required for this disclosure.

[0259] It will be understood that the quantization parameter here can be a pre-set quantization parameter (i.e., an offline quantization parameter) or a quantization parameter obtained based on the processing of the data to be quantized (i.e., an online quantization parameter). During the reasoning, training, and fine-tuning process of the neural network, the target data can be quantized offline or online. Among them, offline quantization can be offline processing of the data to be quantized using the quantization parameter. Online quantization can be online processing of the data to be quantized using the quantization parameter. For example, when the neural network runs on an artificial intelligence chip, the data to be quantized and the quantization parameter can be sent to a computing device outside the artificial intelligence chip for offline quantization, or the pre-obtained data to be quantized and the quantization parameter can be offline quantized using a computing device outside the artificial intelligence chip. In the process of the artificial intelligence chip running the neural network, the artificial intelligence chip can perform online quantization of the data to be quantized using the quantization parameter. When the neural network includes multiple layers to be quantized, each layer to be quantized can be quantized online and offline respectively. In this disclosure, there is no limitation on whether the quantization process of each layer to be quantized is online or offline.

[0260] According to an embodiment of the present disclosure, a method for adjusting data bit width is provided. Figures 8B to 8V Provide a description. Figure 8B A flow chart of a data bit width adjustment method 800B according to an embodiment of the present disclosure is shown. The method may include:

[0261] S114. Determine a quantization error based on the data to be quantized of the current test iteration and the quantized data of the current test iteration, wherein the quantized data of the current test iteration is obtained by quantizing the data to be quantized of the current test iteration.

[0262] Optionally, the processor may quantize the data to be quantized using an initial data bit width to obtain the quantized data. The initial data bit width of the current test iteration may be a hyperparameter, or may be determined based on the data to be quantized in a previous test iteration before the current test iteration.

[0263] Specifically, the processor may determine intermediate representation data based on the data to be quantized in the current test iteration and the quantized data in the current test iteration. Optionally, the intermediate representation data is consistent with the representation format of the aforementioned data to be quantized. For example, the processor may perform inverse quantization on the aforementioned quantized data to obtain intermediate representation data consistent with the representation format of the data to be quantized, where inverse quantization refers to the inverse process of quantization. For example, the quantized data may be obtained using Equation 23. The processor may also perform inverse quantization on the quantized data according to Equation 24 to obtain corresponding intermediate representation data, and determine the quantization error based on the data to be quantized and the intermediate representation data.

[0264]

[0265]

[0266] Furthermore, the processor can calculate the quantization error based on the data to be quantized and its corresponding intermediate representation data. Assume that the data to be quantized in the current test iteration is F x =[z1,z2...,z m ], the intermediate representation data corresponding to the data to be quantized is F x1 =[z1 (n) , z2 (n) ..., z m (n) ]. The processor can calculate the quantized data F x and its corresponding intermediate representation data F x1 An error term is determined, and a quantization error is determined based on the error term.

[0267] Optionally, the processor can x1 The sum of the elements in and the data to be quantized F x The sum of the elements in determines the above error term, which can be the intermediate representation data F x1 The sum of the elements in and the data to be quantized F x The difference between the sum of the elements in . Then, the processor can determine the quantization error based on the error term. The specific quantization error can be determined according to the following formula:

[0268]

[0269] Among them, z i is the element in the data to be quantified, z i (n) is the intermediate representation data F x1 elements.

[0270] Optionally, the processor may calculate the relationship between each element in the data to be quantized and the intermediate representation data F x1 The difference between the corresponding elements in , m differences are obtained, and the sum of the m differences is used as the error term. Afterwards, the processor can determine the quantization error based on the error term. The specific quantization error can be determined according to the following formula:

[0271]

[0272] Among them, z i is the element in the data to be quantified, z i (n) is the intermediate representation data F x1 elements.

[0273] Optionally, each element in the above-mentioned data to be quantized and the intermediate representation data F x1 The difference between the corresponding elements in can be approximately equal to 2 s-1 , therefore, the above quantization error can also be determined according to the following formula:

[0274]

[0275] Among them, m is the intermediate representation data F corresponding to the target data x1 The number of points, s is the point position, z i is the element in the data to be quantified.

[0276] Optionally, the intermediate representation data may also be consistent with the data representation format of the quantized data, and the quantization error may be determined based on the intermediate representation data and the quantized data. For example, the data to be quantized may be represented as: x ≈I x ×2 s , we can determine the intermediate representation data The intermediate representation data I x1 The data representation format may be the same as the above-mentioned quantized data. x1 And the above formula (23) calculated Determine the quantization error. The specific method for determining the quantization error can refer to the above formulas (25) to (27).

[0277] S115. Determine a target data bit width corresponding to the current test iteration according to the quantization error.

[0278] Specifically, the processor can adaptively adjust the data bit width corresponding to the current test iteration based on the quantization error, and determine the target data bit width after adjustment of the current test iteration. When the quantization error meets the preset conditions, the data bit width corresponding to the current test iteration can be kept unchanged, that is, the target data bit width of the current test iteration can be equal to the initial data bit width. When the quantization error does not meet the preset conditions, the processor can adjust the data bit width corresponding to the data to be quantized in the current test iteration to obtain the target data bit width corresponding to the current test iteration. When the processor uses the target data bit width to quantize the data to be quantized in the current test iteration, the quantization error meets the above-mentioned preset conditions. Optionally, the above-mentioned preset conditions can be a preset threshold set by the user.

[0279] Optionally, Figure 8C FIG. 8 is a flow chart of a data bit width adjustment method 800C according to another embodiment of the present disclosure. Figure 8C As shown, the above operation S115 may include:

[0280] S1150: The processor may determine whether the quantization error is greater than or equal to a first preset threshold.

[0281] If the quantization error is greater than or equal to the first preset threshold, operation S1151 may be performed to increase the data bit width corresponding to the current test iteration to obtain the target data bit width of the current test iteration. If the quantization error is less than the first preset threshold, the data bit width of the current test iteration may be kept unchanged.

[0282] Further optionally, the processor may obtain the aforementioned target data bit width through a single adjustment. For example, if the initial data bit width of the current test iteration is n1, the processor may determine the target data bit width n2 = n1 + t through a single adjustment, where t is the adjusted value of the data bit width. When the target data bit width n2 is used to quantize the data to be quantized in the current test iteration, the resulting quantization error may be less than the first preset threshold.

[0283] Further optionally, the processor may obtain the target data bit width through multiple adjustments until the quantization error is less than a first preset threshold, and use the data bit width when the quantization error is less than the first preset threshold as the target data bit width. Specifically, if the quantization error is greater than or equal to the first preset threshold, the first intermediate data bit width is determined according to the first preset bit width step; the processor may then quantize the data to be quantized in the current test iteration according to the first intermediate data bit width to obtain quantized data, and determine the quantization error based on the data to be quantized in the current test iteration and the quantized data of the current test iteration until the quantization error is less than the first preset threshold. The processor may use the data bit width corresponding to the quantization error being less than the first preset threshold as the target data bit width.

[0284] For example, if the initial data bit width of the current test iteration is n1, the processor may use this initial data bit width n1 to quantize the data to be quantized A for the current test iteration to obtain quantized data B1, and calculate a quantization error C1 based on the data to be quantized A and the quantized data B1. When the quantization error C1 is greater than or equal to a first preset threshold, the processor determines a first intermediate data bit width n2 = n1 + t1, where t1 is a first preset bit width step size. The processor may then quantize the data to be quantized for the current test iteration based on the first intermediate data bit width n2 to obtain quantized data B2 for the current test iteration, and calculate a quantization error C2 based on the data to be quantized A and the quantized data B2. If the quantization error C2 is greater than or equal to the first preset threshold, the processor may determine a first intermediate data bit width n2 = n1 + t1 + t1, and then quantize the data to be quantized A for the current test iteration based on this new first intermediate data bit width and calculate the corresponding quantization error until the quantization error is less than the first preset threshold. If the quantization error C1 is less than the first preset threshold, the initial data bit width n1 may be maintained unchanged.

[0285] Furthermore, the first preset bit width step size may be a constant value. For example, whenever the quantization error is greater than a first preset threshold, the processor may increase the data bit width corresponding to the current test iteration by the same bit width value. Alternatively, the first preset bit width step size may be a variable value. For example, the processor may calculate the difference between the quantization error and the first preset threshold. The smaller the difference between the quantization error and the first preset threshold, the smaller the value of the first preset bit width step size.

[0286] Optionally, Figure 8D FIG. 8 is a flow chart of a data bit width adjustment method 800D according to another embodiment of the present disclosure. Figure 8D As shown, the above operation S115 may further include:

[0287] S1152: The processor may determine whether the quantization error is less than or equal to a second preset threshold.

[0288] If the quantization error is less than or equal to the second preset threshold, operation S1153 may be performed to reduce the data bit width corresponding to the current test iteration to obtain the target data bit width of the current test iteration. If the quantization error is greater than the second preset threshold, the data bit width of the current test iteration may be kept unchanged.

[0289] Further optionally, the processor may obtain the aforementioned target data bit width through a single adjustment. For example, if the initial data bit width of the current test iteration is n1, the processor may determine the target data bit width n2 = n1 - t through a single adjustment, where t is the adjusted value of the data bit width. When the target data bit width n2 is used to quantize the data to be quantized in the current test iteration, the resulting quantization error may be greater than the second preset threshold.

[0290] Further optionally, the processor may obtain the target data bit width through multiple adjustments until the quantization error is greater than a second preset threshold, and use the data bit width when the quantization error is greater than the second preset threshold as the target data bit width. Specifically, if the quantization error is less than or equal to the first preset threshold, the second intermediate data bit width is determined based on the second preset bit width step; the processor may then quantize the data to be quantized in the current test iteration based on the second intermediate data bit width to obtain quantized data, and determine the quantization error based on the data to be quantized in the current test iteration and the quantized data of the current test iteration until the quantization error is greater than the second preset threshold. The processor may use the data bit width corresponding to the quantization error being greater than the second preset threshold as the target data bit width.

[0291] For example, if the initial data bit width of the current test iteration is n1, the processor may use this initial data bit width n1 to quantize the data to be quantized A for the current test iteration to obtain quantized data B1, and calculate a quantization error C1 based on the data to be quantized A and the quantized data B1. When the quantization error C1 is less than or equal to a second preset threshold, the processor determines a second intermediate data bit width n2 = n1 - t2, where t2 is the second preset bit width step size. The processor may then quantize the data to be quantized for the current test iteration based on the second intermediate data bit width n2 to obtain quantized data B2 for the current test iteration, and calculate a quantization error C2 based on the data to be quantized A and the quantized data B2. If the quantization error C2 is less than or equal to the second preset threshold, the processor may determine a second intermediate data bit width n2 = n1 - t2 - t2, and then quantize the data to be quantized A for the current test iteration based on this new second intermediate data bit width and calculate the corresponding quantization error until the quantization error exceeds the second preset threshold. If the quantization error C1 is greater than the second preset threshold, the initial data bit width n1 may remain unchanged.

[0292] Furthermore, the second preset bit width step size may be a constant value. For example, whenever the quantization error is less than the second preset threshold, the processor may reduce the data bit width corresponding to the current test iteration by the same bit width value. Alternatively, the second preset bit width step size may be a variable value. For example, the processor may calculate the difference between the quantization error and the second preset threshold. The smaller the difference between the quantization error and the second preset threshold, the smaller the value of the second preset bit width step size.

[0293] Optionally, Figure 8E FIG. 8 is a flow chart of a data bit width adjustment method 800E according to another embodiment of the present disclosure. Figure 8E As shown, when the processor determines that the quantization error is less than the first preset threshold and the quantization error is greater than the second preset threshold, the data bit width of the current test iteration can be kept unchanged, wherein the first preset threshold is greater than the second preset threshold. That is, the target data bit width of the current test iteration can be equal to the initial data bit width. Figure 8E The method for determining the data bit width of one embodiment of the present disclosure is described in the following by way of example only. Figure 8E The order of each operation can be adjusted adaptively and is not specifically limited here.

[0294] Figure 8F FIG. 8 is a flow chart of a neural network quantization method 800F according to an embodiment of the present disclosure. Figure 8F As shown, the neural network quantization method includes:

[0295] Step S10: determining a quantization parameter corresponding to each type of data to be quantized in the layer to be quantized, where the data to be quantized includes at least one of neurons, weights, biases, and gradients.

[0296] In step S20 , the data to be quantized is quantized according to corresponding quantization parameters to obtain quantized data, so that the neural network performs operations according to the quantized data.

[0297] Step S30 : determining a quantization error of the target data according to target data and quantization data corresponding to the target data, wherein the target data is any type of data to be quantized.

[0298] The quantization error of the target data can be determined based on the error between the quantized data corresponding to the target data and the target data. The quantization error of the target data can be calculated using a predetermined error calculation method, such as a standard deviation calculation method, a root mean square error calculation method, etc.

[0299] Alternatively, the quantized data corresponding to the target data may be dequantized according to the quantization parameter to obtain dequantized data, and the quantization error of the target data may be determined according to the error between the dequantized data and the target data.

[0300] When the quantization parameter includes the point position, the quantized data of the target data can be dequantized according to formula (28) to obtain the dequantized data of the target data

[0301]

[0302] Among them: round is the rounding operation, is the inverse quantized data of the target data, and s is the point position corresponding to the target data.

[0303] When the quantization parameter includes a scaling factor, the quantized data of the target data can be dequantized according to formula (29) to obtain the dequantized data of the target data.

[0304]

[0305] Among them: round is the rounding operation. is the inverse quantized data of the target data, and f is the scaling factor.

[0306] When the quantization parameter includes an offset, the quantized data of the target data can be dequantized according to formula (30) to obtain the dequantized data of the target data

[0307]

[0308] Among them: round is the rounding operation. is the inverse quantized data of the target data, and o is the scaling factor.

[0309] When the quantization parameters include point positions and scaling coefficients, the quantized data of the target data can be dequantized according to formula (31) to obtain the dequantized data of the target data.

[0310]

[0311] When the quantization parameters include point position and offset, the quantized data of the target data can be dequantized according to formula (32) to obtain the dequantized data of the target data

[0312]

[0313] When the quantization parameter includes a scaling factor and an offset, the quantized data of the target data can be dequantized according to formula (33) to obtain the dequantized data of the target data

[0314]

[0315] When the quantization parameters include point position, scaling factor and offset, the quantized data of the target data can be dequantized according to formula (34) to obtain the dequantized data of the target data

[0316]

[0317] The error diff between the target data and the inverse quantized data corresponding to the target data can be obtained by calculating the quantization interval correlation method, for example, by using formula (35) bit :

[0318]

[0319] Where p is the number of elements in the target data, and s is the point position of the target data. The value of A can be determined according to the quantization parameter. When the quantization parameter includes the point position s, A=2 s ; When the quantization parameters include the point position s and the scaling factor f, A = 2 s ×f.

[0320] Alternatively, the difference between the means of the two data can be calculated, for example, by using formula (36) to calculate the error diff between the target data and the inverse quantized data corresponding to the target data: bit :

[0321]

[0322] Alternatively, the mean of the difference between the two data can be calculated, for example, by using formula (37) to calculate the error diff between the target data and the inverse quantized data corresponding to the target data: bit :

[0323]

[0324] Step S40 : adjusting the data bit width corresponding to the target data according to the quantization error and the error threshold to obtain an adjusted bit width corresponding to the target data.

[0325] An error threshold can be determined based on empirical values and can be used to represent the expected value of the quantization error. When the quantization error is greater than or less than the error threshold, the data bit width corresponding to the target number can be adjusted to obtain an adjusted bit width corresponding to the target data. The data bit width can be adjusted to a longer or shorter bit width to increase or decrease quantization accuracy.

[0326] The error threshold can be determined based on the maximum acceptable error. When the quantization error is greater than the error threshold, it indicates that the quantization accuracy cannot meet expectations and the data bit width needs to be adjusted to a longer bit width. Alternatively, a smaller error threshold can be determined based on a higher quantization accuracy. When the quantization error is less than the error threshold, it indicates that the quantization accuracy is high and the operating efficiency of the neural network will be affected. The data bit width can be appropriately adjusted to a shorter bit width to appropriately reduce the quantization accuracy and improve the operating efficiency of the neural network.

[0327] The data bit width may be adjusted according to a fixed bit step length, or may be adjusted according to a variable adjustment step length based on the difference between the quantization error and the error threshold, which is not limited in this disclosure.

[0328] Step S50: updating the data bit width corresponding to the target data to the adjusted bit width, and calculating a corresponding adjusted quantization parameter according to the target data and the adjusted bit width, so that the neural network is quantized according to the adjusted quantization parameter.

[0329] After determining the adjusted bit width, the data bit width corresponding to the target data can be updated to the adjusted bit width. For example, if the data bit width of the target data before the update is 8 bits and the adjusted bit width is 12 bits, the data bit width corresponding to the updated target data will be 12 bits. The adjusted quantization parameter corresponding to the target data can be calculated based on the adjusted bit width and the target data. The target data can be requantized based on the adjusted quantization parameter corresponding to the target data to obtain quantized data with higher or lower quantization accuracy, so that the quantization layer achieves a balance between quantization accuracy and processing efficiency.

[0330] During the reasoning, training, and fine-tuning of a neural network, the data to be quantized between layers can be considered to have a certain correlation. For example, when the difference between the mean values of the data to be quantized between layers is less than a set mean threshold, and the difference between the maximum values of the data to be quantized between layers is also less than a set difference threshold, the adjusted quantization parameters of the layer to be quantized can be used as the adjusted quantization parameters of one or more subsequent layers to quantize the data to be quantized in one or more subsequent layers of the layer to be quantized. During the training and fine-tuning of a neural network, the adjusted quantization parameters of the layer to be quantized obtained in the current iteration can also be used to quantize the layer to be quantized in subsequent iterations.

[0331] In a possible implementation, the method further includes:

[0332] One or more layers following the layer to be quantized adopt the quantization parameter of the layer to be quantized.

[0333] The neural network performs quantization according to the adjusted quantization parameter, which may include re-quantizing the data to be quantized using the adjusted quantization parameter only in the layer to be quantized, and using the re-obtained quantized data for operations in the layer to be quantized. It may also include re-quantizing the data to be quantized without using the adjusted quantization parameter in the layer to be quantized, and using the adjusted quantization parameter for quantization in one or more layers subsequent to the layer to be quantized, and / or using the adjusted quantization parameter for quantization in subsequent iterations. It may also include re-quantizing using the adjusted quantization parameter in the layer to be quantized, and using the re-obtained quantized data for operations in the layer to be quantized, and using the adjusted quantization parameter for quantization in one or more layers subsequent to the layer to be quantized, and / or using the adjusted quantization parameter for quantization in subsequent iterations. This disclosure is not limited to this.

[0334] In this embodiment, the quantization error of the target data is determined based on the target data and the quantization data corresponding to the target data, where the target data is any type of data to be quantized; the data bit width corresponding to the target data is adjusted using the quantization error and the error threshold to obtain an adjusted bit width corresponding to the target data; the data bit width corresponding to the target data is updated to the adjusted bit width, and a corresponding adjusted quantization parameter is calculated based on the target data and the adjusted bit width, so that the neural network performs quantization based on the adjusted quantization parameter. The data bit width is adjusted based on the error between the target data and the quantization data, and the adjusted quantization parameter is calculated based on the adjusted data bit width. Different adjusted quantization parameters can be obtained by setting different error thresholds to achieve different quantization requirements such as improving quantization accuracy or improving operating efficiency. The adjusted quantization parameter calculated based on the target data and the quantization data of the target data can also better conform to the data characteristics of the target data itself, achieving a quantization result that better meets the requirements of the target data itself, and achieving a better balance between quantization accuracy and processing efficiency.

[0335] Figure 8G FIG. 8 is a flow chart of a neural network quantization method 800G according to an embodiment of the present disclosure. Figure 8G As shown, step S40 in the neural network quantization method includes:

[0336] Step S41 : when the quantization error is greater than the first error threshold, increasing the data bit width corresponding to the target data to obtain an adjusted bit width corresponding to the target data.

[0337] A first error threshold can be determined based on a maximum acceptable quantization error. The quantization error can be compared with the first error threshold. When the quantization error is greater than the first error threshold, the quantization error can be considered unacceptable. To improve quantization accuracy, the quantization accuracy of the target data can be increased by increasing the data bit width corresponding to the target data.

[0338] The data bit width corresponding to the target data can be increased by a fixed adjustment step size to obtain an adjusted bit width. The fixed adjustment step size can be N bits, where N is a positive integer. Each adjustment step can increase the data bit width by N bits. The data bit width after each increase = the original data bit width + N bits.

[0339] The data bit width corresponding to the target data can be increased according to a variable adjustment step to obtain an adjusted bit width. For example, when the difference between the quantization error and the error threshold is greater than the first threshold, the data bit width can be adjusted according to the adjustment step M1. When the difference between the quantization error and the error threshold is less than the first threshold, the data bit width can be adjusted according to the adjustment step M2, wherein the first threshold is greater than the second threshold, and M1 is greater than M2. Each variable adjustment step can be determined according to demand. This disclosure does not limit the adjustment step of the data bit width and whether the adjustment step is variable.

[0340] The target data can be calculated according to the adjusted bit width to obtain the adjusted quantization parameter. The quantized data obtained by re-quantizing the target data using the adjusted quantization parameter has a higher quantization accuracy than the quantized data obtained by quantizing using the quantization parameter before the adjustment.

[0341] Figure 8H FIG. 8 is a flow chart of a neural network quantization method 800H according to an embodiment of the present disclosure. Figure 8H As shown, the neural network quantization method further includes:

[0342] Step S42, calculating the adjusted quantization error of the target data according to the adjusted bit width and the target data;

[0343] Step S43 : Continue to increase the adjusted bit width according to the adjusted quantization error and the first error threshold until the adjusted quantization error calculated according to the adjusted bit width and the target data is less than or equal to the first error threshold.

[0344] When the data bit width corresponding to the target data is increased according to the quantization error, the bit width is adjusted once to obtain the adjusted bit width, the adjusted quantization parameter is calculated based on the adjusted bit width, the target data is quantized according to the adjusted quantization parameter to obtain the adjusted quantized data, and the adjusted quantization error of the target data is calculated based on the adjusted quantized data and the target data. The adjusted quantization error may still be greater than the first error threshold, that is, the data bit width adjusted once may not meet the adjustment purpose. If the adjusted quantization error is still greater than the first error threshold, the adjusted data bit width can be further adjusted, that is, the data bit width corresponding to the target data can be increased multiple times until the adjusted quantization error obtained based on the final adjusted bit width and the target data is less than the first error threshold.

[0345] The adjustment step size for multiple increases can be fixed or variable. For example, the final data width = original data width + A*N bits, where N is the fixed adjustment step size for each increase and A is the number of times the data width is increased. The final data width = original data width + M1 + M2 + ... + Mm, where M1, M2, ..., Mm are the variable adjustment step sizes for each increase.

[0346] In this embodiment, when the quantization error is greater than a first error threshold, the data bit width corresponding to the target data is increased to obtain an adjusted bit width corresponding to the target data. The data bit width can be increased by setting the first error threshold and the adjustment step size so that the adjusted data bit width meets the quantization requirements. If a single adjustment cannot meet the adjustment requirements, the data bit width can be adjusted multiple times. Setting the first error threshold and the adjustment step size allows the quantization parameters to be flexibly adjusted according to the quantization requirements, meeting different quantization requirements and enabling the quantization accuracy to be adaptively adjusted based on the data characteristics.

[0347] Figure 8I FIG. 8 is a flow chart of a neural network quantization method 800I according to an embodiment of the present disclosure. Figure 8I As shown, step S40 in the neural network quantization method includes:

[0348] Step S44 : when the quantization error is less than the second error threshold, reducing the data bit width corresponding to the target data, and the second error threshold is less than the first error threshold.

[0349] A second error threshold can be determined based on an acceptable quantization error and the desired operating efficiency of the neural network. The quantization error can be compared with the second error threshold. When the quantization error is less than the second error threshold, it can be considered that the quantization error exceeds expectations, but the operating efficiency is too low to be acceptable. The quantization accuracy can be reduced to improve the operating efficiency of the neural network. The quantization accuracy of the target data can be reduced by reducing the data bit width corresponding to the target data.

[0350] The data bit width corresponding to the target data can be reduced by a fixed adjustment step size to obtain an adjusted bit width. The fixed adjustment step size can be N bits, where N is a positive integer. Each adjustment reduces the data bit width by N bits. The increased data bit width = the original data bit width - N bits.

[0351] The data bit width corresponding to the target data can be reduced according to a variable adjustment step to obtain an adjusted bit width. For example, when the difference between the quantization error and the error threshold is greater than the first threshold, the data bit width can be adjusted according to the adjustment step M1. When the difference between the quantization error and the error threshold is less than the first threshold, the data bit width can be adjusted according to the adjustment step M2, wherein the first threshold is greater than the second threshold, and M1 is greater than M2. Each variable adjustment step can be determined according to demand. This disclosure does not limit the adjustment step of the data bit width and whether the adjustment step is variable.

[0352] The target data may be calculated according to the adjusted bit width to obtain an adjusted quantization parameter. The quantized data obtained by re-quantizing the target data using the adjusted quantization parameter has lower quantization accuracy than the quantized data obtained by quantizing using the quantization parameter before the adjustment.

[0353] Figure 8J FIG. 8 is a flow chart of a neural network quantization method 800J according to an embodiment of the present disclosure. Figure 8J As shown, the neural network quantization method further includes:

[0354] Step S45, calculating the adjusted quantization error of the target data according to the adjusted bit width and the target data;

[0355] Step S46 : Continue reducing the adjusted bit width according to the adjusted quantization error and the second error threshold until the adjusted quantization error calculated according to the adjusted bit width and the target data is greater than or equal to the second error threshold.

[0356] When the data bit width corresponding to the target data is increased according to the quantization error, the bit width is adjusted once to obtain an adjusted bit width, an adjusted quantization parameter is calculated based on the adjusted bit width, the target data is quantized according to the adjusted quantization parameter to obtain adjusted quantized data, and then an adjusted quantization error of the target data is calculated based on the adjusted quantized data and the target data. The adjusted quantization error may still be less than the second error threshold, that is, the data bit width adjusted once may not meet the adjustment purpose. When the adjusted quantization error is still less than the second error threshold, the adjusted data bit width can be further adjusted, that is, the data bit width corresponding to the target data can be reduced multiple times until the adjusted quantization error obtained based on the final adjusted bit width and the target data is greater than the second error threshold.

[0357] The adjustment step size for multiple reductions can be fixed or variable. For example, the final data bit width = original data bit width - A*N bits, where N is the fixed adjustment step size for each increase and A is the number of times the data bit width is increased. The final data bit width = original data bit width - M1 - M2 - ... - Mm, where M1, M2, ..., Mm are the variable adjustment step sizes for each reduction.

[0358] In this embodiment, when the quantization error is less than the second error threshold, the data bit width corresponding to the target data is reduced to obtain the adjusted bit width corresponding to the target data. The data bit width can be reduced by setting the second error threshold and the adjustment step size so that the adjusted data bit width can meet the quantization requirements. When one adjustment cannot meet the adjustment requirements, the data bit width can be adjusted multiple times. The setting of the second error threshold and the adjustment step size allows the quantization parameters to be flexibly and adaptively adjusted according to the quantization requirements, meeting different quantization requirements, making the quantization accuracy adjustable, and achieving a balance between the quantization accuracy and the operating efficiency of the neural network.

[0359] In a possible implementation, the method further includes:

[0360] When the quantization error is greater than a first error threshold, the data bit width corresponding to the target data is increased, and when the quantization error is less than a second error threshold, the data bit width corresponding to the target data is reduced to obtain an adjusted bit width corresponding to the target data.

[0361] Alternatively, two error thresholds can be set simultaneously, where the first error threshold indicates that the quantization accuracy is too low and the number of bits in the data bit width can be increased, while the second error threshold indicates that the quantization accuracy is too high and the number of bits in the data bit width can be reduced. If the first error threshold is greater than the second error threshold, the quantization error of the target data can be compared with both error thresholds simultaneously. When the quantization error is greater than the first error threshold, the number of bits in the data bit width is increased, and when the quantization error is less than the second error threshold, the number of bits in the data bit width is reduced. When the quantization error is between the first and second error thresholds, the data bit width can remain unchanged.

[0362] In this embodiment, by comparing the quantization error with both the first error threshold and the second error threshold, the data bit width can be increased or decreased based on the comparison results. The first error threshold and the second error threshold can be used to more flexibly adjust the data bit width, so that the data bit width adjustment result better meets the quantization requirements.

[0363] It should be understood that neural network training refers to the process of performing multiple iterative operations on a neural network (the weights of which may be random numbers) so that the weights of the neural network meet preset conditions. One iterative operation generally includes a forward operation, a reverse operation, and a weight update operation. The forward operation refers to the process of performing forward inference based on the input data of the neural network to obtain a forward operation result. The reverse operation refers to the process of determining a loss value based on the forward operation result and a preset reference value, and then determining the weight gradient value and / or the input data gradient value based on the loss value. The weight update operation refers to the process of adjusting the weights of the neural network based on the weight gradient value. Specifically, the neural network training process is as follows: a processor may perform a forward operation on the input data using a neural network with random weights to obtain a forward operation result. The processor then determines a loss value based on the forward operation result and a preset reference value, and then determines the weight gradient value and / or the input data gradient value based on the loss value. Finally, the processor may update the gradient value of the neural network based on the weight gradient value to obtain new weights, completing one iterative operation. The processor loops and executes multiple iterations until the forward operation result of the neural network meets a preset condition. For example, training ends when the forward operation result of the neural network converges to a preset reference value. Alternatively, training ends when the loss value determined by the forward operation result of the neural network and the preset reference value is less than or equal to a preset accuracy.

[0364] Fine-tuning is the process of performing multiple iterations on a neural network (whose weights have converged rather than being random numbers) to ensure that the neural network's accuracy meets preset requirements. This fine-tuning process is essentially the same as the training process described above and can be considered the process of retraining a neural network that has already converged. Inference is the process of performing forward operations on a neural network whose weights meet preset conditions to achieve functions such as recognition or classification, such as using a neural network for image recognition.

[0365] In the disclosed embodiments, during the training or fine-tuning of the neural network, different quantization parameters can be used at different stages of the neural network operation to quantize the neural network operation data, and iterative operations can be performed based on the quantized data, thereby reducing the data storage space during the neural network operation process and improving data access efficiency and operation efficiency. Figure 8K FIG. 8 is a flowchart of a quantization parameter adjustment method 800K according to an embodiment of the present disclosure. The method may include:

[0366] S100: Obtain the data variation range of the data to be quantified.

[0367] Optionally, the processor may directly read the data variation range of the data to be quantified, and the data variation range of the data to be quantified may be input by a user.

[0368] Optionally, the processor may also calculate the data variation amplitude of the data to be quantified based on the data to be quantified in the current iteration and the data to be quantified in previous iterations, where the current iteration refers to the currently executed iterative operation, and the previous iteration refers to the iterative operation executed before the current iteration. For example, the processor may obtain the maximum value and average value of the elements in the data to be quantified in the current iteration, as well as the maximum value and average value of the elements in the data to be quantified in each previous iteration, and determine the variation amplitude of the data to be quantified based on the maximum value and average value of the elements in each iteration. If the maximum value of the elements in the data to be quantified in the current iteration is close to the maximum value of the elements in the data to be quantified in a preset number of previous iterations, and the average value of the elements in the data to be quantified in the current iteration is close to the average value of the elements in the data to be quantified in a preset number of previous iterations, then the data variation amplitude of the data to be quantified may be determined to be small. Otherwise, the data variation amplitude of the data to be quantified may be determined to be large. For another example, the data variation amplitude of the data to be quantified may be represented by a sliding average or variance of the data to be quantified, etc., which is not specifically limited here.

[0369] In the disclosed embodiments, the magnitude of the data fluctuation of the data to be quantized can be used to determine whether the quantization parameters of the data to be quantized need to be adjusted. For example, if the magnitude of the data fluctuation of the data to be quantized is large, it can be indicated that the quantization parameters need to be adjusted promptly to ensure quantization accuracy. If the magnitude of the data fluctuation of the data to be quantized is small, the quantization parameters of the previous iterations can be used for the current test iteration and a certain number of iterations thereafter, thereby avoiding frequent adjustments to the quantization parameters and improving quantization efficiency.

[0370] Each iteration involves at least one piece of data to be quantized, which can be floating-point or fixed-point data. Optionally, the data to be quantized in each iteration can be at least one of neuron data, weight data, or gradient data. The gradient data can also include neuron gradient data and weight gradient data.

[0371] S200. Determine a target iteration interval based on the data variation of the data to be quantized, and adjust a quantization parameter in a neural network operation based on the target iteration interval. The target iteration interval includes at least one iteration, and the quantization parameter of the neural network is used to implement a quantization operation on the data to be quantized in the neural network operation. The quantization parameter may include data width. Therefore, the target iteration interval may be determined based on the data variation of the data to be quantized, and the data width in the neural network operation may be adjusted based on the target iteration interval. The target iteration interval includes at least one iteration.

[0372] Optionally, the quantization parameter may include the above-mentioned point position and / or scaling factor, wherein the scaling factor may include a first scaling factor and a second scaling factor. The specific point position calculation method and scaling factor calculation method can refer to the formula described above and will not be repeated here. Optionally, the quantization parameter may also include an offset, and the calculation method of the offset can refer to the formula described above; further, the processor may also determine the point position and the scaling factor according to other formulas described above. In the embodiment of the present disclosure, the processor may update at least one of the above-mentioned point position, scaling factor or offset according to the determined target iteration interval to adjust the quantization parameter in the neural network operation. That is to say, the quantization parameter in the neural network operation can be updated according to the data variation amplitude of the data to be quantized in the neural network operation, so as to ensure the quantization accuracy.

[0373] It is understandable that by statistically analyzing the changing trend of the computational data during the training or fine-tuning of the neural network, a data change curve of the data to be quantified can be obtained. Figure 8L As shown, according to the data change curve 800L, it can be seen that in the early stage of neural network training or fine-tuning, the data changes of the data to be quantized at different iterations are relatively drastic. As the training or fine-tuning operation proceeds, the data changes of the data to be quantized at different iterations gradually become gentle. Therefore, in the early stage of neural network training or fine-tuning, the quantization parameters can be adjusted more frequently; in the middle and late stages of neural network training or fine-tuning, the quantization parameters can be adjusted after multiple iterations or training cycles. The method disclosed herein is to achieve a balance between quantization accuracy and quantization efficiency by determining the appropriate iteration interval.

[0374] Specifically, the processor can determine the target iteration interval based on the data variation amplitude of the data to be quantized, so as to adjust the quantization parameter in the neural network operation according to the target iteration interval. Optionally, the target iteration interval can increase as the data variation amplitude of the data to be quantized decreases. That is to say, the larger the data variation amplitude of the data to be quantized, the smaller the target iteration interval, indicating that the adjustment of the quantization parameter is more frequent. The smaller the data variation amplitude of the data to be quantized, the larger the target iteration interval, indicating that the adjustment of the quantization parameter is less frequent. Of course, in other embodiments, the above-mentioned target iteration interval can also be a hyperparameter. For example, the target iteration interval can be a user-defined setting.

[0375] Optionally, the various data to be quantized, such as the weight data, neuron data, and gradient data, may each have a different iteration interval. Accordingly, the processor may respectively obtain the data variation amplitude corresponding to each type of data to be quantized, and determine the target iteration interval corresponding to the corresponding type of data to be quantized based on the data variation amplitude of each type of data to be quantized. In other words, the quantization process of the various types of data to be quantized may be performed asynchronously. In the disclosed embodiment, since there are differences between different types of data to be quantized, different data variation amplitudes of the data to be quantized may be used to determine the corresponding target iteration interval, and the corresponding quantization parameters may be determined based on the corresponding target iteration interval, thereby ensuring the quantization accuracy of the data to be quantized and further ensuring the correctness of the calculation results of the neural network.

[0376] Of course, in other embodiments, the same target iteration interval can be determined for different types of data to be quantized, and the quantization parameters corresponding to the data to be quantized can be adjusted based on the target iteration interval. For example, the processor can obtain the data variation amplitudes of various types of data to be quantized, determine the target iteration interval based on the largest data variation amplitude of the data to be quantized, and determine the quantization parameters of the various types of data to be quantized based on the target iteration interval. Furthermore, the same quantization parameters can be used for different types of data to be quantized.

[0377] Further optionally, the neural network may include at least one computation layer, and the data to be quantized may be at least one of neuron data, weight data, or gradient data involved in each computation layer. In this case, the processor may obtain the data to be quantized involved in the current computation layer and determine the data variation range and corresponding target iteration interval for each type of data to be quantized in the current computation layer according to the above method.

[0378] Optionally, the processor may determine the data variation amplitude of the data to be quantized once during each iterative operation, and determine a target iteration interval based on the corresponding data variation amplitude of the data to be quantized. In other words, the processor may calculate the target iteration interval once for each iteration. The specific method for calculating the target iteration interval is described below. Furthermore, the processor may select a test iteration from each iteration based on preset conditions, determine the variation amplitude of the data to be quantized at each test iteration, and update and adjust the quantization parameters, etc. based on the target iteration interval corresponding to the test iteration. In this case, if the iteration is not the selected test iteration, the processor may ignore the target iteration interval corresponding to the iteration.

[0379] Optionally, each target iteration interval may correspond to a test iteration, and the test iteration may be the starting iteration of the target iteration interval or the ending iteration of the target iteration interval. The processor may adjust the quantization parameter of the neural network at the test iteration of each target iteration interval to adjust the quantization parameter of the neural network operation according to the target iteration interval. The test iteration may be a time point for verifying whether the current quantization parameter meets the requirements of the data to be quantized. The quantization parameter before adjustment may be the same as the quantization parameter after adjustment, or may be different from the quantization parameter after adjustment. Optionally, the interval between adjacent test iterations may be greater than or equal to one target iteration interval.

[0380] For example, the target iteration interval can calculate the number of iterations starting from the current test iteration, and the current test iteration can be the starting iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 3 iterations, namely the 100th iteration, the 101st iteration, and the 102nd iteration. The processor can adjust the quantization parameter in the neural network operation at the 100th iteration. Among them, the current test iteration is the iterative operation corresponding to the quantization parameter update adjustment currently performed by the processor.

[0381] Optionally, the target iteration interval may also be a number of iterations calculated starting from the next iteration of the current test iteration, where the current test iteration may be the termination iteration of the previous iteration interval before the current test iteration. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation amplitude of the data to be quantized. The processor may determine that the target iteration interval includes 3 iterations, namely the 101st iteration, the 102nd iteration, and the 103rd iteration. The processor may adjust the quantization parameters in the neural network operation at the 100th iteration and the 103rd iteration. The present disclosure does not specifically limit the method for determining the target iteration interval.

[0382] In one embodiment, according to the calculation formulas of the midpoint position, scaling factor and offset described above, it can be seen that the quantization parameter is often related to the data to be quantized. Therefore, in the above operation S100, the data variation range of the data to be quantized can also be indirectly determined by the variation range of the quantization parameter. The data variation range of the data to be quantized can be characterized by the variation range of the quantization parameter. Specifically, Figure 8M FIG. 8 is a flow chart showing a method 800M for determining a target iteration interval in a parameter adjustment method according to an embodiment of the present disclosure. The operation S100 may include:

[0383] S110 , obtaining a variation range of a point position; wherein the variation range of the point position can be used to characterize a data variation range of the data to be quantified, and the variation range of the point position is positively correlated with the data variation range of the data to be quantified.

[0384] Optionally, the amplitude of the point position change can indirectly reflect the amplitude of the data to be quantified. The amplitude of the point position change can be determined based on the point position of the current test iteration and the point position of at least one historical iteration. The point position of the current test iteration and the point positions of each historical iteration can be determined using the formula described above.

[0385] For example, the processor may also calculate the variance of the point position of the current test iteration and the point position of the historical iteration, and determine the variation range of the point position based on the variance. For another example, the processor may determine the variation range of the point position based on the average value of the point position of the current test iteration and the point position of the historical iteration. Specifically, Figure 8N As shown, Figure 8N The flowchart of the method 800N for determining the variation range of a point position in one embodiment of the present disclosure is shown. The above operation S110 may include:

[0386] S111: Determine a first mean value based on point positions corresponding to a previous test iteration before the current test iteration and point positions corresponding to a historical iteration before the previous test iteration, where the previous test iteration is an iteration corresponding to a previous adjustment of the quantization parameter, and the previous test iteration is separated from the current test iteration by at least one iteration interval.

[0387] Optionally, at least one historical iteration may belong to at least one iteration interval, each iteration interval may correspond to one test iteration, and two adjacent test iterations may have one iteration interval. The previous test iteration in the above operation S111 may be the test iteration corresponding to the previous iteration interval before the target iteration interval.

[0388] Optionally, the first mean value may be calculated according to the following formula:

[0389] M1=a1×s t-1 +a2×s t-2 +a3×s t-3 +…+am×s 1 Formula (38)

[0390] Among them, a1~am refers to the calculation weight corresponding to the point position of each iteration, s t-1 Refers to the point position corresponding to the previous test iteration, s t-2 , s t-3 ...s 1 refers to the point position corresponding to the historical iteration preceding the previous test iteration, and M1 refers to the first mean mentioned above. Furthermore, based on the data distribution characteristics, the farther away a historical iteration is from the previous test iteration, the less impact it has on the distribution and fluctuation of point positions in iterations near the previous test iteration. Therefore, the above calculation weights can be reduced in the order of a1 to am.

[0391] For example, if the last test iteration is the 100th iteration of the neural network operation, and the historical iterations may be the 1st iteration to the 99th iteration, the processor can obtain the point position of the 100th iteration (i.e., s t-1 ), and obtain the point position of the historical iteration before the 100th iteration, that is, s 1 It can refer to the point position corresponding to the first iteration of the neural network..., s t-3 It can refer to the point position corresponding to the 98th iteration of the neural network, s t-2 It may refer to the point position corresponding to the 99th iteration of the neural network. Further, the processor may calculate and obtain the first mean value according to the above formula.

[0392] Furthermore, the first mean can be calculated based on the point positions of the test iterations corresponding to each iteration interval. For example, the first mean can be calculated according to the following formula:

[0393] M1=a1×s t-1 +a2×s t-2 +a3×s t-3 +…+am×s 1 ;

[0394] Among them, a1~am refers to the calculation weight corresponding to the point position of each test iteration, s t-1 Refers to the point position corresponding to the previous test iteration, s t-2 , s t-3 ...s 1 It refers to the point position corresponding to the test iteration of the preset number of iteration intervals before the previous test iteration, and M1 refers to the first mean mentioned above.

[0395] For example, the last test iteration is the 100th iteration of the neural network operation, and the historical iterations can be the 1st iteration to the 99th iteration. The 99 historical iterations can be divided into 11 iteration intervals. For example, the 1st iteration to the 9th iteration belong to the 1st iteration interval, the 10th iteration to the 18th iteration belong to the 2nd iteration interval, ..., the 90th iteration to the 99th iteration belong to the 11th iteration interval. Then the processor can obtain the point position of the 100th iteration (i.e., s t-1 ), and obtain the point position of the test iteration in the iteration interval before the 100th iteration, that is, s 1 It can refer to the point position corresponding to the test iteration of the first iteration interval of the neural network (such as s 1 It can refer to the point position corresponding to the first iteration of the neural network), ..., s t-3 It can refer to the point position corresponding to the test iteration of the 10th iteration interval of the neural network (for example, s t-3 It can refer to the point position corresponding to the 81st iteration of the neural network), s t-2 It can refer to the point position corresponding to the test iteration of the 11th iteration interval of the neural network (for example, s t-2 (may refer to the point position corresponding to the 90th iteration of the neural network). Further, the processor may calculate and obtain the first mean value M1 according to the above formula.

[0396] In the disclosed embodiments, for ease of illustration, it is assumed that the iteration intervals contain the same number of iterations. However, in actual use, the number of iterations contained in the iteration intervals may vary. Alternatively, the number of iterations contained in the iteration intervals increases as the number of iterations increases, that is, the iteration intervals may become larger as the neural network training or fine-tuning progresses.

[0397] Furthermore, in order to further simplify the calculation and reduce the storage space occupied by the data, the first mean M1 can be calculated using the following formula:

[0398] M1=α×s t-1 +(1-α)×M0 Formula (39)

[0399] Among them, α refers to the calculation weight of the point position corresponding to the previous test iteration, s t-1 It refers to the point position corresponding to the previous test iteration, and M0 refers to the sliding average corresponding to the test iteration before the previous test iteration. The specific calculation method of M0 can refer to the calculation method of M1 mentioned above, which will not be repeated here.

[0400] S112: Determine a second mean value based on the point position corresponding to the current test iteration and the point position of the historical iteration before the current test iteration. The point position corresponding to the current test iteration can be determined based on the target data bit width of the current test iteration and the data to be quantized.

[0401] Optionally, the second mean M2 may be calculated according to the following formula:

[0402] M2=b1×s t +b2×s t-1 +b3×s t-2 +…+bm×s 1 Formula(40)

[0403] Among them, b1~bm refer to the calculation weights corresponding to the point positions of each iteration, s t Refers to the point position corresponding to the current test iteration, s t-1 , s t-2 ...s 1 refers to the point position corresponding to the historical iteration preceding the current test iteration, and M2 refers to the aforementioned second mean. Furthermore, based on the data distribution characteristics, the farther away a historical iteration is from the current test iteration, the less influence it has on the distribution and fluctuation of point positions in iterations near the current test iteration. Therefore, the aforementioned calculation weights can be reduced in the order of b1 to bm.

[0404] For example, the current test iteration is the 101th iteration of the neural network operation, and the historical iterations before the current test iteration are the 1st iteration to the 100th iteration. The processor can obtain the point position of the 101st iteration (i.e., s t ), and obtain the point position of the historical iteration before the 101th iteration, that is, s 1 It can refer to the point position corresponding to the first iteration of the neural network..., s t-2 It can refer to the point position corresponding to the 99th iteration of the neural network, s t-1 It may refer to the point position corresponding to the 100th iteration of the neural network. Further, the processor may calculate and obtain the second mean value M2 according to the above formula.

[0405] Optionally, the second mean can be calculated based on the point positions of the test iterations corresponding to each iteration interval. Specifically, Figure 8O FIG. 8 is a flow chart showing a method 8000 for determining a second mean value in accordance with an embodiment of the present disclosure. The operation S112 may include the following operations:

[0406] S1121. Obtain a preset number of intermediate sliding averages, wherein each intermediate sliding average is determined based on a preset number of test iterations before the current test iteration, the test iteration being an iteration corresponding to adjusting parameters in the neural network quantization process;

[0407] S1122. Determine the second mean according to the point position of the current inspection iteration and the preset number of intermediate sliding averages.

[0408] For example, the second mean can be calculated according to the following formula:

[0409] M2=b1×s t +b2×s t-1 +b3×s t-2 +…+bm×s 1 ;

[0410] Among them, b1~bm refer to the calculation weights corresponding to the point positions of each iteration, s t Refers to the point position corresponding to the current test iteration, s t-1 , s t-2 ...s 1 It refers to the point position corresponding to the test iteration before the current test iteration, and M2 refers to the second mean mentioned above.

[0411] For example, the current test iteration is the 100th iteration, and the historical iterations may be the 1st iteration to the 99th iteration. The 99 historical iterations may belong to 11 iteration intervals. For example, the 1st iteration to the 9th iteration belong to the 1st iteration interval, the 10th iteration to the 18th iteration belong to the 2nd iteration interval, and so on, the 90th iteration to the 99th iteration belong to the 11th iteration interval. Then the processor can obtain the point position of the 100th iteration (i.e., s t ), and obtain the point position of the test iteration in the iteration interval before the 100th iteration, that is, s 1 It can refer to the point position corresponding to the test iteration of the first iteration interval of the neural network (such as s 1 It can refer to the point position corresponding to the first iteration of the neural network), ..., s t-2 It can refer to the point position corresponding to the test iteration of the 10th iteration interval of the neural network (for example, s t-2 It can refer to the point position corresponding to the 81st iteration of the neural network), s t-1 It can refer to the point position corresponding to the test iteration of the 11th iteration interval of the neural network (for example, s t-1 (may refer to the point position corresponding to the 90th iteration of the neural network). Further, the processor can calculate and obtain the second mean value M2 according to the above formula.

[0412] In the disclosed embodiments, for ease of illustration, it is assumed that the iteration intervals contain the same number of iterations. However, in actual use, the number of iterations contained in the iteration intervals may vary. Alternatively, the number of iterations contained in the iteration intervals increases as the number of iterations increases, that is, the iteration intervals may become larger as the neural network training or fine-tuning progresses.

[0413] Furthermore, to simplify calculation and reduce storage space occupied by data, the processor may determine the second mean value based on the point position corresponding to the current test iteration and the first mean value. That is, the second mean value may be calculated using the following formula:

[0414] M2=β×s t +(1-β)×M1 Formula (41)

[0415] Among them, β refers to the calculation weight of the point position corresponding to the current test iteration, and M1 refers to the first mean mentioned above.

[0416] S113. Determine a first error based on the first mean and the second mean, where the first error is used to characterize a variation range of the point positions of the current inspection iteration and the historical iteration.

[0417] Optionally, the first error may be equal to the absolute value of the difference between the second mean and the first mean. Specifically, the first error may be calculated according to the following formula:

[0418] diff update1 =|M2-M1|=β|s (t) -M1| Formula (42)

[0419] Optionally, the point position of the current test iteration can be determined based on the data to be quantized in the current test iteration and the target data bit width corresponding to the current test iteration. The specific point position calculation method can be determined by referring to the formula above. Among them, the target data bit width corresponding to the current test iteration can be a hyperparameter. Further optionally, the target data bit width corresponding to the current test iteration can be user-defined input. Optionally, the data bit width corresponding to the data to be quantized during the training or fine-tuning of the neural network can be constant, that is, the same type of data to be quantized in the same neural network is quantized using the same data bit width. For example, the neuron data in each iteration of the neural network are quantized using a data bit width of 8 bits.

[0420] Optionally, the data bit width corresponding to the data to be quantized during neural network training or fine-tuning is variable to ensure that the data bit width meets the quantization requirements of the data to be quantized. In other words, the processor can adaptively adjust the data bit width corresponding to the data to be quantized based on the data to be quantized to obtain a target data bit width corresponding to the data to be quantized. Specifically, the processor can first determine the target data bit width corresponding to the current test iteration, and then the processor can determine the point position corresponding to the current test iteration based on the target data bit width corresponding to the current test iteration and the data to be quantized corresponding to the current test iteration.

[0421] In the disclosed embodiment, when the data bit width of the current test iteration changes, the point position will change accordingly. However, the change in the point position is not caused by the data change of the data to be quantized. The target iteration interval obtained by calculating the first error determined by the above formula (42) may be inaccurate, which will affect the accuracy of quantization. Therefore, when the data bit width of the current test iteration changes, the above second mean can be adjusted accordingly to ensure that the first error can accurately reflect the change amplitude of the point position, thereby ensuring the accuracy and reliability of the target iteration interval. Specifically, if Figure 8P FIG. 8 is a flow chart showing a method 800P for determining a second mean value in another embodiment of the present disclosure. The method may further include:

[0422] S116. Determine a data bit width adjustment value for the current inspection iteration according to the target data bit width;

[0423] Specifically, the processor can determine the data width adjustment value for the current test iteration based on the target data width and the initial data width for the current test iteration. Here, the data width adjustment value = the target data width - the initial data width. Of course, the processor can also directly obtain the data width adjustment value for the current test iteration.

[0424] S117: Update the second mean value according to the data width adjustment value of the current test iteration.

[0425] Specifically, if the data width adjustment value is greater than a preset parameter (for example, the preset parameter may be equal to zero), that is, when the data width of the current test iteration increases, the processor may reduce the second mean accordingly. If the data width adjustment value is less than the preset parameter (for example, the preset parameter may be equal to zero), that is, when the data width of the current test iteration decreases, the processor may increase the second mean accordingly. If the data width adjustment value is equal to the preset parameter, that is, when the data width adjustment value is equal to 0, the data to be quantized corresponding to the current iteration has not changed, and the updated second mean is equal to the second mean before the update, and the second mean before the update is calculated according to the above formula (41). Optionally, if the data width adjustment value is equal to the preset parameter, that is, when the data width adjustment value is equal to 0, the processor may not update the second mean, that is, the processor may not perform the above operation S117.

[0426] For example, the second mean before updating M2 = β × s t +(1-β)×M1; when the target data width n2 corresponding to the current test iteration = initial data width n1+Δn, where Δn represents the data width adjustment value. At this time, the updated second mean M2 = β×(s t -Δn)+(1-β)×(M1-Δn). When the target data width n2 corresponding to the current test iteration = the initial data width n1-Δn, where Δn represents the data width adjustment value, the updated second mean M2 = β×(s t -Δn)+(1-β)×(M1+Δn), where s t It means that the current test iteration is based on the target data bit width to determine the point position.

[0427] For example, the second mean before update M2 = β × s t +(1-β)×M1; when the target data width n2 corresponding to the current test iteration = the initial data width n1 + Δn, where Δn represents the data width adjustment value. At this time, the updated second mean M2 = β×s t +(1-β)×M1-Δn. For another example, when the target data width n2 corresponding to the current test iteration = the initial data width n1-Δn, where Δn represents the data width adjustment value, the updated second mean M2 = β×s t +(1-β)×M1+Δn, where s t It means that the current test iteration is based on the target data bit width to determine the point position.

[0428] Furthermore, the operation S200 may include:

[0429] The target iteration interval is determined based on the amplitude of the point position fluctuation. The target iteration interval is negatively correlated with the amplitude of the point position fluctuation. That is, the larger the amplitude of the point position fluctuation, the smaller the target iteration interval. The smaller the amplitude of the point position fluctuation, the larger the target iteration interval.

[0430] As described above, the first error may represent the variation range of the point position, and the above operations may include:

[0431] The processor may determine the target iteration interval based on the first error, wherein the target iteration interval is negatively correlated with the first error. Specifically, a larger first error indicates a larger amplitude of change in point position, and thus a larger amplitude of change in the data to be quantized. In this case, the target iteration interval is smaller.

[0432] Specifically, the processor can calculate the target iteration interval I according to the following formula:

[0433]

[0434] Among them, I is the target iteration interval, diff update1 represents the first error mentioned above, and δ and γ can be hyperparameters.

[0435] It is understood that the first error can be used to measure the fluctuation range of the point position. The larger the first error, the greater the fluctuation range of the point position, which in turn indicates that the greater the fluctuation range of the data to be quantized, the smaller the target iteration interval needs to be set. In other words, the larger the first error, the more frequent the quantization parameter adjustment.

[0436] In this embodiment, the target iteration interval is determined based on the variation in the point positions (the first error) calculated. Since the quantization parameter is determined based on the target iteration interval, the quantized data obtained by quantization based on the quantization parameter is more consistent with the variation trend of the point positions of the target data, thereby improving the operating efficiency of the neural network while ensuring quantization accuracy.

[0437] Optionally, after determining the target iteration interval at the current test iteration, the processor may further determine parameters such as the quantization parameter and data bit width corresponding to the target iteration interval at the current test iteration, thereby updating the quantization parameter according to the target iteration interval. The quantization parameter may include a point position and / or a scaling factor. Furthermore, the quantization parameter may also include an offset. The specific calculation method of the quantization parameter can be found in the description above. Figure 8Q FIG. 8 is a flowchart of a quantization parameter adjustment method 800Q according to another embodiment of the present disclosure. The method may further include:

[0438] S300: The processor adjusts the quantization parameters in the neural network operation according to the target iteration interval.

[0439] Specifically, the processor can determine the test iteration based on the target iteration interval, update the target iteration interval at each test iteration, and can also update the quantization parameters at each test iteration. For example, if the data bit width in the neural network operation remains unchanged, the processor can adjust the quantization parameters such as the point position at each test iteration directly based on the data to be quantized in the test iteration. For another example, if the data bit width in the neural network operation is variable, the processor can update the data bit width at each test iteration and adjust the quantization parameters such as the point position based on the updated data bit width and the data to be quantized in the test iteration.

[0440] In the disclosed embodiment, the processor updates the quantization parameter at each test iteration to ensure that the current quantization parameter meets the quantization requirements of the data to be quantized. The target iteration interval before the update and the target iteration interval after the update may be the same or different. The data bit width before the update and the data bit width after the update may be the same or different; that is, the data bit widths at different iteration intervals may be the same or different. The quantization parameter before the update and the quantization parameter after the update may be the same or different; that is, the quantization parameters at different iteration intervals may be the same or different.

[0441] Optionally, in the above operation S300, the processor may determine a quantization parameter in a target iteration interval at the test iteration to adjust the quantization parameter in the neural network operation.

[0442] In one scenario, the data bit width corresponding to each iteration of the neural network operation does not change, that is, the data bit width corresponding to each iteration of the neural network operation is the same. In this case, the processor can achieve the purpose of adjusting the quantization parameters in the neural network operation according to the target iteration interval by determining the quantization parameters such as the point position in the target iteration interval. The quantization parameters corresponding to the iterations in the target iteration interval can be consistent. In other words, each iteration in the target iteration interval uses the same point position, and only the quantization parameters such as the point position are updated at each test iteration. This avoids updating and adjusting the quantization parameters for each iteration, thereby reducing the amount of computation in the quantization process and improving the efficiency of the quantization operation.

[0443] Optionally, for the case where the data bit width remains unchanged, the point position corresponding to the iteration in the target iteration interval can remain consistent. Specifically, the processor can determine the point position corresponding to the current test iteration based on the data to be quantized in the current test iteration and the target data bit width corresponding to the current test iteration, and use the point position corresponding to the current test iteration as the point position corresponding to the target iteration interval. The iterations in the target iteration interval all use the point position corresponding to the current test iteration. Optionally, the target data bit width corresponding to the current test iteration can be a hyperparameter. For example, the target data bit width corresponding to the current test iteration is input by a user. The point position corresponding to the current test iteration can be calculated with reference to the formula above.

[0444] In one case, the data bit width corresponding to each iteration in the neural network operation may change, that is, the data bit width corresponding to different target iteration intervals may be inconsistent, but the data bit width of each iteration in the target iteration interval remains unchanged. The data bit width corresponding to the iteration in the target iteration interval may be a hyperparameter, for example, the data bit width corresponding to the iteration in the target iteration interval may be user-defined input. In one case, the data bit width corresponding to the iteration in the target iteration interval may also be obtained by calculation by the processor, for example, the processor may determine the target data bit width corresponding to the current test iteration based on the data to be quantized in the current test iteration, and use the target data bit width corresponding to the current test iteration as the data bit width corresponding to the target iteration interval.

[0445] To simplify the computational effort during the quantization process, the corresponding quantization parameters, such as the point positions, can remain unchanged within the target iteration interval. In other words, the same point positions are used for each iteration within the target iteration interval, and only the quantization parameters, such as the point positions, and the data bit width are updated at each test iteration. This avoids the need to update and adjust the quantization parameters at each iteration, reducing the computational effort during the quantization process and improving the efficiency of the quantization operation.

[0446] Optionally, for the case where the data bit width corresponding to the above-mentioned target iteration interval remains unchanged, the point position corresponding to the iteration in the target iteration interval can remain consistent. Specifically, the processor can determine the point position corresponding to the current test iteration based on the data to be quantized in the current test iteration and the target data bit width corresponding to the current test iteration, and use the point position corresponding to the current test iteration as the point position corresponding to the target iteration interval. The iterations in the target iteration interval all use the point position corresponding to the current test iteration. Optionally, the target data bit width corresponding to the current test iteration can be a hyperparameter. For example, the target data bit width corresponding to the current test iteration is input by a user. The point position corresponding to the current test iteration can be calculated with reference to the formula above.

[0447] Optionally, the scaling coefficients corresponding to iterations in the target iteration interval may be consistent. The processor may determine the scaling coefficient corresponding to the current test iteration based on the to-be-quantized data of the current test iteration, and use the scaling coefficient corresponding to the current test iteration as the scaling coefficient for each iteration in the target iteration interval. The scaling coefficients corresponding to iterations in the target iteration interval are consistent.

[0448] Optionally, the offsets corresponding to iterations in the target iteration interval are consistent. The processor can determine the offset corresponding to the current test iteration based on the data to be quantized in the current test iteration, and use the offset corresponding to the current test iteration as the offset for each iteration in the target iteration interval. Furthermore, the processor can also determine the minimum and maximum values of all elements of the data to be quantized, and further determine quantization parameters such as point positions and scaling factors, as described above. The offsets corresponding to iterations in the target iteration interval are consistent.

[0449] For example, the target iteration interval can calculate the number of iterations starting from the current test iteration, that is, the test iteration corresponding to the target iteration interval can be the starting iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 3 iterations, namely the 100th iteration, the 101st iteration, and the 102nd iteration. Then, the processor can determine the quantization parameters such as the point position corresponding to the 100th iteration based on the data to be quantized and the target data bit width corresponding to the 100th iteration, and can use the quantization parameters such as the point position corresponding to the 100th iteration to quantize the 100th iteration, the 101st iteration, and the 102nd iteration. In this way, the processor does not need to calculate the quantization parameters such as the point position in the 101st iteration and the 102nd iteration, which reduces the amount of calculation in the quantization process and improves the efficiency of the quantization operation.

[0450] Optionally, the target iteration interval may also be calculated starting from the next iteration of the current test iteration, that is, the test iteration corresponding to the target iteration interval may also be the terminating iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation amplitude of the data to be quantized. The processor can then determine that the target iteration interval includes 3 iterations, namely the 101st iteration, the 102nd iteration, and the 103rd iteration. Furthermore, the processor can determine the quantization parameters such as the point position corresponding to the 100th iteration based on the data to be quantized and the target data bit width corresponding to the 100th iteration, and can use the quantization parameters such as the point position corresponding to the 100th iteration to quantize the 101st iteration, the 102nd iteration, and the 103rd iteration. In this way, the processor does not need to calculate the quantization parameters such as the point position in the 102nd iteration and the 103rd iteration, thereby reducing the amount of calculation in the quantization process and improving the efficiency of the quantization operation.

[0451] In the disclosed embodiments, the data bit width and quantization parameters corresponding to each iteration within the same target iteration interval are consistent. That is, the data bit width, point position, scaling factor, and offset corresponding to each iteration within the same target iteration interval remain unchanged. This avoids frequent adjustments to the quantization parameters of the data to be quantized during neural network training or fine-tuning, reduces the amount of computation required during the quantization process, and thus improves quantization efficiency. Furthermore, by dynamically adjusting the quantization parameters based on the magnitude of data fluctuations at different stages of training or fine-tuning, quantization accuracy can be guaranteed.

[0452] In another case, the data bit width corresponding to each iteration in the neural network operation may change, but the data bit width of each iteration in the target iteration interval remains unchanged. At this time, the quantitative parameters such as the point position corresponding to the iteration in the target iteration interval may also be inconsistent. The processor can also determine the data bit width corresponding to the target iteration interval based on the target data bit width corresponding to the current test iteration, wherein the data bit width corresponding to the iteration in the target iteration interval is consistent. Afterwards, the processor can adjust the quantitative parameters such as the point position in the neural network operation process based on the data bit width corresponding to the target iteration interval and the point position iteration interval. Optionally, as Figure 8R FIG. 8 is a flowchart of a method 800R for adjusting a quantization parameter in a quantization parameter adjustment method according to an embodiment of the present disclosure. The operation S300 may further include:

[0453] S310. Determine the data bit width corresponding to the target iteration interval based on the data to be quantized in the current test iteration; wherein the data bit width corresponding to each iteration in the target iteration interval is consistent. In other words, the data bit width during the neural network operation is updated every target iteration interval. Optionally, the data bit width corresponding to the target iteration interval may be the target data bit width for the current test iteration. The target data bit width for the current test iteration can be described in operations S114 and S115 above and will not be further described here.

[0454] For example, the target iteration interval can calculate the number of iterations starting from the current test iteration, that is, the test iteration corresponding to the target iteration interval can be the starting iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 6 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 6 iterations, namely the 100th iteration to the 105th iteration. At this time, the processor can determine the target data bit width of the 100th iteration, and the target data bit width of the 100th iteration is used from the 101st iteration to the 105th iteration. There is no need to calculate the target data bit width from the 101st iteration to the 105th iteration, thereby reducing the amount of calculation and improving the quantization efficiency and the operation efficiency. Afterwards, the 106th iteration can be used as the current test iteration, and the above-mentioned operations of determining the target iteration interval and updating the data bit width are repeated.

[0455] Optionally, the target iteration interval can also be calculated by starting from the next iteration of the current test iteration, that is, the test iteration corresponding to the target iteration interval can also be the termination iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 6 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 6 iterations, namely the 101st iteration to the 106th iteration. At this time, the processor can determine the target data bit width of the 100th iteration, and the 101st iteration to the 106th iteration continue to use the target data bit width of the 100th iteration, and there is no need to calculate the target data bit width from the 101st iteration to the 106th iteration, thereby reducing the amount of calculation and improving the quantization efficiency and operation efficiency. Afterwards, the 106th iteration can be used as the current test iteration, and the above-mentioned operations of determining the target iteration interval and updating the data bit width are repeated.

[0456] S320. The processor adjusts the point position corresponding to the iteration in the target iteration interval according to the obtained point position iteration interval and the data bit width corresponding to the target iteration interval, so as to adjust quantitative parameters such as the point position in the neural network operation.

[0457] The point position iteration interval includes at least one iteration, and the iterated point positions in the point position iteration interval are consistent. Optionally, the point position iteration interval can be a hyperparameter, for example, the point position iteration interval can be user-defined.

[0458] Optionally, the point position iteration interval is less than or equal to the target iteration interval. When the point position iteration interval is the same as the above-mentioned target iteration interval, the processor can synchronously update the data bit width, point position and other quantization parameters at the current test iteration. Further optionally, the scaling coefficients corresponding to the iterations in the target iteration interval can be consistent. Furthermore, the offsets corresponding to the iterations in the target iteration interval are consistent. At this time, the data bit width, point position and other quantization parameters corresponding to the iterations in the target iteration interval are the same, thereby reducing the amount of calculation and improving quantization efficiency and operational efficiency. The specific implementation process is basically the same as that of the above-mentioned embodiment, and reference can be made to the description above, which will not be repeated here.

[0459] When the point position iteration interval is less than the target iteration interval, the processor may update the data bit width, point position, and other quantization parameters at the test iteration corresponding to the target iteration interval, and may also update the point position and other quantization parameters at the sub-test iteration determined by the point position iteration interval. Since the point position and other quantization parameters can be fine-tuned based on the data to be quantized while the data bit width remains unchanged, the point position and other quantization parameters can also be adjusted within the same target iteration interval to further improve quantization accuracy.

[0460] Specifically, the processor may determine a sub-test iteration based on the current test iteration and the point position iteration interval. The sub-test iteration is used to adjust the point position. The sub-test iteration may be an iteration within the target iteration interval. Furthermore, the processor may adjust the point position corresponding to the iteration within the target iteration interval based on the to-be-quantized data of the sub-test iteration and the data bit width corresponding to the target iteration interval. The method for determining the point position can refer to the above formula and is not further described here.

[0461] For example, the current test iteration is the 100th iteration, the target iteration interval is 6, and the iterations included in the target iteration interval are the 100th iteration to the 105th iteration. The point position iteration interval obtained by the processor is I s1 =3, the point position can be adjusted every three iterations starting from the current test iteration. Specifically, the processor can use the 100th iteration as the above-mentioned sub-test iteration, and calculate the point position s1 corresponding to the 100th iteration. The point position s1 is shared and quantized in the 100th, 101st, and 102nd iterations. Afterwards, the processor can adjust the point position according to the point position iteration interval I s1The 103rd iteration is used as the aforementioned sub-test iteration. The processor can also determine the point position s2 corresponding to the second point position iteration interval based on the data to be quantized corresponding to the 103rd iteration and the data bit width n corresponding to the target iteration interval. The point position s2 can then be shared for quantization in iterations 103 through 105. In the disclosed embodiment, the values of the point position s1 before the update and the point position s2 after the update can be the same or different. Furthermore, the processor can re-determine the next target iteration interval and the corresponding quantization parameters, such as the data bit width and point position, based on the data variation range of the data to be quantized, at iteration 106.

[0462] For another example, the current test iteration is the 100th iteration, the target iteration interval is 6, and the iterations included in the target iteration interval are the 101st iteration to the 106th iteration. s1 =3, then the point position can be adjusted every three iterations starting from the current test iteration. Specifically, the processor can determine that the point position corresponding to the first point position iteration interval is s1 based on the data to be quantized in the current test iteration and the target data bit width n1 corresponding to the current test iteration. Then, the point position s1 is shared for quantization in the 101st, 102nd and 103rd iterations. Afterwards, the processor can adjust the point position according to the point position iteration interval I s1 The 104th iteration is used as the aforementioned sub-test iteration. The processor can also determine the point position s2 corresponding to the second point position iteration interval based on the data to be quantized corresponding to the 104th iteration and the data bit width n1 corresponding to the target iteration interval. The point position s2 can then be shared for quantization in iterations 104 through 106. In the disclosed embodiment, the values of the point position s1 before the update and the point position s2 after the update can be the same or different. Furthermore, the processor can re-determine the next target iteration interval and the corresponding quantization parameters, such as the data bit width and point position, based on the data variation range of the data to be quantized, at iteration 106.

[0463] Optionally, the point position iteration interval may be equal to 1, meaning that the point position is updated once for each iteration. Optionally, the point position iteration intervals may be the same or different. For example, at least one point position iteration interval included in the target iteration interval may increase sequentially. This is merely an example to illustrate the implementation of this embodiment and is not intended to limit the present disclosure.

[0464] Optionally, the scaling factors corresponding to iterations within the target iteration interval may be different. Further, the scaling factor may be updated synchronously with the point position, i.e., the iteration interval corresponding to the scaling factor may be equal to the point position iteration interval. That is, each time the processor updates the determined point position, the scaling factor is updated accordingly.

[0465] Optionally, the offsets corresponding to the iterations in the target iteration interval may also be inconsistent. Furthermore, the offset may be updated synchronously with the above-mentioned point position, that is, the iteration interval corresponding to the offset may be equal to the above-mentioned point position iteration interval. That is, whenever the processor updates the determined point position, the determined offset will be updated accordingly. Of course, the offset may also be updated asynchronously with the above-mentioned location position or data bit width, which is not specifically limited here. Furthermore, the processor may also determine the minimum and maximum values of all elements of the data to be quantized, and further determine quantization parameters such as point position and scaling factor, for details see the description above.

[0466] In another embodiment, the processor can comprehensively determine the data variation range of the data to be quantized based on the variation range of the point position and the change in the data bit width of the data to be quantized, and determine the target iteration interval based on the data variation range of the data to be quantized, wherein the target iteration interval can be used to update the determined data bit width, that is, the processor can update the determined data bit width at the test iteration of each target iteration interval. Since the point position can reflect the accuracy of the fixed-point data and the data bit width can reflect the data representation range of the fixed-point data, by comprehensively considering the variation range of the point position and the change in the data bit width of the data to be quantized, it can be ensured that the quantized data can take into account both the accuracy and the data representation range. Optionally, the variation range of the point position can be characterized by the above-mentioned first error, and the change in the data bit width can be determined based on the above-mentioned quantization error. Specifically, as Figure 8S FIG. 8 is a flow chart showing a method 800S for determining a target iteration interval in a parameter adjustment method according to another embodiment of the present disclosure. The method may include:

[0467] S400. Obtain a first error. The first error can characterize the variation range of the point position. The variation range of the point position can represent the data variation range of the data to be quantized. Specifically, the calculation method of the above-mentioned first error can refer to the description in operation S110 above, which will not be repeated here.

[0468] S500: Obtain a second error, where the second error is used to characterize a change in the data bit width.

[0469] Optionally, the second error can be determined based on the quantization error, and the second error is positively correlated with the quantization error. Figure 8TFIG. 8 is a flow chart showing a method 800T for determining a target iteration interval in a parameter adjustment method according to another embodiment of the present disclosure. The operation S500 may include:

[0470] S510: Determine a quantization error based on the data to be quantized in the current test iteration and the quantized data of the current test iteration, wherein the quantized data of the current test iteration is obtained by quantizing the data to be quantized in the current test iteration according to the initial data bit width. The specific method for determining the quantization error can be found in the description of operation S114 above and is not further described here.

[0471] S520: Determine the second error according to the quantization error, where the second error is positively correlated with the quantization error.

[0472] Specifically, the second error can be calculated according to the following formula:

[0473] diff update2 =θ*diff bit 2 Formula (44)

[0474] Among them, diff update2 Represents the second error mentioned above, diff bit represents the above quantization error, and θ can be a hyperparameter.

[0475] return Figure 8S , at S600, determine the target iteration interval based on the second error and the first error.

[0476] Specifically, the processor can calculate a target error based on the first error and the second error, and determine a target iteration interval based on the target error. Alternatively, the target error can be calculated by taking a weighted average of the first error and the second error. For example, target error = K * first error + (1-K) * second error, where K is a hyperparameter. The processor can then determine a target iteration interval based on the target error, where the target iteration interval is negatively correlated with the target error. That is, the larger the target error, the smaller the target iteration interval.

[0477] Optionally, the target error can also be determined based on the maximum value of the first error and the second error, in which case the weight of the first error or the second error is 0. Specifically, Figure 8T As shown, the above operation S600 may include:

[0478] S610: Taking the maximum value of the first error and the second error as the target error.

[0479] Specifically, the processor may compare the first error diff update1 and the second error diffupdate2 The size of the first error diff update1 Greater than the second error diff update2 When , the target error is equal to the first error diff update1 When the first error diff update1 When it is less than the second error, the target error is equal to the second error diff update2 When the first error diff update1 When it is equal to the second error, the target error can be the first error diff update1 or the second error diff update2 . That is, the target error diff update It can be determined according to the following formula:

[0480] diff update =max(diff update1 , diff update2 ) Formula (45)

[0481] Among them, diff update Refers to the target error, diff update1 Refers to the first error, diff update2 It refers to the second error.

[0482] S620: Determine the target iteration interval based on the target error, wherein the target error is negatively correlated with the target iteration interval. Specifically, the target iteration interval can be determined as follows:

[0483] The target iteration interval can be calculated according to the following formula:

[0484]

[0485] Among them, I represents the target iteration interval, diff update represents the target error mentioned above, and δ and γ can be hyperparameters.

[0486] Optionally, in the above embodiment, the data bit width in the neural network operation is variable, and the variation trend of the data bit width can be measured by the second error. Figure 8TAs shown, after determining the target iteration interval, the processor can perform operation S630 to determine the data bit width corresponding to the iteration in the target iteration interval, wherein the data bit width corresponding to the iteration in the target iteration interval is consistent. Specifically, the processor can determine the data bit width corresponding to the target iteration interval based on the data to be quantized in the current test iteration. In other words, the data bit width in the neural network operation process is updated once every target iteration interval. Optionally, the data bit width corresponding to the target iteration interval can be the target data bit width of the current test iteration. The target data bit width of the current test iteration can be referred to operations S114 and S115 above, which will not be repeated here.

[0487] For example, the target iteration interval can calculate the number of iterations starting from the current test iteration, that is, the test iteration corresponding to the target iteration interval can be the starting iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 6 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 6 iterations, namely the 100th iteration to the 105th iteration. At this time, the processor can determine the target data bit width of the 100th iteration, and the target data bit width of the 100th iteration is used from the 101st iteration to the 105th iteration. There is no need to calculate the target data bit width from the 101st iteration to the 105th iteration, thereby reducing the amount of calculation and improving the quantization efficiency and the operation efficiency. Afterwards, the 106th iteration can be used as the current test iteration, and the above-mentioned operations of determining the target iteration interval and updating the data bit width are repeated.

[0488] Optionally, the target iteration interval can also be calculated by starting from the next iteration of the current test iteration, that is, the test iteration corresponding to the target iteration interval can also be the termination iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 6 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 6 iterations, namely the 101st iteration to the 106th iteration. At this time, the processor can determine the target data bit width of the 100th iteration, and the 101st iteration to the 106th iteration continue to use the target data bit width of the 100th iteration, and there is no need to calculate the target data bit width from the 101st iteration to the 106th iteration, thereby reducing the amount of calculation and improving the quantization efficiency and operation efficiency. Afterwards, the 106th iteration can be used as the current test iteration, and the above-mentioned operations of determining the target iteration interval and updating the data bit width are repeated.

[0489] Furthermore, the processor can also determine the quantization parameters in the target iteration interval at the test iteration, and adjust the quantization parameters in the neural network operation according to the target iteration interval. That is, the quantization parameters such as the point position in the neural network operation can be updated synchronously with the data bit width.

[0490] In one embodiment, the quantization parameters corresponding to iterations in the target iteration interval can be consistent. Optionally, the processor can determine the point position corresponding to the current test iteration based on the to-be-quantized data of the current test iteration and the target data bit width corresponding to the current test iteration, and use the point position corresponding to the current test iteration as the point position corresponding to the target iteration interval, wherein the point positions corresponding to iterations in the target iteration interval are consistent. In other words, each iteration in the target iteration interval uses the quantization parameters, such as the point position, of the current test iteration, avoiding the need to update and adjust the quantization parameters for each iteration, thereby reducing the computational complexity of the quantization process and improving the efficiency of the quantization operation.

[0491] Optionally, the scaling coefficients corresponding to iterations in the target iteration interval may be consistent. The processor may determine the scaling coefficient corresponding to the current test iteration based on the to-be-quantized data of the current test iteration, and use the scaling coefficient corresponding to the current test iteration as the scaling coefficient for each iteration in the target iteration interval. The scaling coefficients corresponding to iterations in the target iteration interval are consistent.

[0492] Optionally, the offsets corresponding to iterations in the target iteration interval are consistent. The processor can determine the offset corresponding to the current test iteration based on the data to be quantized in the current test iteration, and use the offset corresponding to the current test iteration as the offset for each iteration in the target iteration interval. Furthermore, the processor can also determine the minimum and maximum values of all elements of the data to be quantized, and further determine quantization parameters such as point positions and scaling factors, as described above. The offsets corresponding to iterations in the target iteration interval are consistent.

[0493] For example, the target iteration interval can calculate the number of iterations starting from the current test iteration, that is, the test iteration corresponding to the target iteration interval can be the starting iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation amplitude of the data to be quantized. Then the processor can determine that the target iteration interval includes 3 iterations, namely the 100th iteration, the 101st iteration, and the 102nd iteration. Then, the processor can determine the quantization parameters such as the point position corresponding to the 100th iteration based on the data to be quantized and the target data bit width corresponding to the 100th iteration, and can use the quantization parameters such as the point position corresponding to the 100th iteration to quantize the 100th iteration, the 101st iteration, and the 102nd iteration. In this way, the processor does not need to calculate the quantization parameters such as the point position in the 101st iteration and the 102nd iteration, which reduces the amount of calculation in the quantization process and improves the efficiency of the quantization operation.

[0494] Optionally, the target iteration interval may also be calculated starting from the next iteration of the current test iteration, that is, the test iteration corresponding to the target iteration interval may also be the terminating iteration of the target iteration interval. For example, the current test iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation amplitude of the data to be quantized. The processor can then determine that the target iteration interval includes 3 iterations, namely the 101st iteration, the 102nd iteration, and the 103rd iteration. Furthermore, the processor can determine the quantization parameters such as the point position corresponding to the 100th iteration based on the data to be quantized and the target data bit width corresponding to the 100th iteration, and can use the quantization parameters such as the point position corresponding to the 100th iteration to quantize the 101st iteration, the 102nd iteration, and the 103rd iteration. In this way, the processor does not need to calculate the quantization parameters such as the point position in the 102nd iteration and the 103rd iteration, thereby reducing the amount of calculation in the quantization process and improving the efficiency of the quantization operation.

[0495] In the disclosed embodiments, the data bit width and quantization parameters corresponding to each iteration within the same target iteration interval are consistent. That is, the data bit width, point position, scaling factor, and offset corresponding to each iteration within the same target iteration interval remain unchanged. This avoids frequent adjustments to the quantization parameters of the data to be quantized during neural network training or fine-tuning, reduces the amount of computation required during the quantization process, and thus improves quantization efficiency. Furthermore, by dynamically adjusting the quantization parameters based on the magnitude of data fluctuations at different stages of training or fine-tuning, quantization accuracy can be guaranteed.

[0496] In another case, the processor may also determine the quantization parameters in the target iteration interval based on the point position iteration interval corresponding to the point position quantization parameters, so as to adjust the quantization parameters in the neural network operation accordingly. That is, the point position quantization parameters in the neural network operation may be updated asynchronously with the data bit width. The processor may update the data bit width and the point position quantization parameters at the test iteration of the target iteration interval. The processor may also separately update the point position corresponding to the iteration in the target iteration interval based on the point position iteration interval.

[0497] Specifically, the processor can also determine the data bit width corresponding to the target iteration interval based on the target data bit width corresponding to the current test iteration, wherein the data bit width corresponding to the iterations in the target iteration interval is consistent. Afterwards, the processor can adjust the quantitative parameters such as the point position in the neural network operation process based on the data bit width corresponding to the target iteration interval and the point position iteration interval. Figure 8TAs shown, after determining the data bit width corresponding to the target iteration interval, the processor may further perform operation S640 to adjust the point positions corresponding to the iterations in the target iteration interval based on the obtained point position iteration interval and the data bit width corresponding to the target iteration interval, thereby adjusting the point positions in the neural network operation. The point position iteration interval includes at least one iteration, and the point positions iterated in the point position iteration interval are consistent. Optionally, the point position iteration interval may be a hyperparameter, for example, the point position iteration interval may be user-defined.

[0498] Optionally, the point position iteration interval is less than or equal to the target iteration interval. When the point position iteration interval is the same as the above-mentioned target iteration interval, the processor can synchronously update the data bit width, point position and other quantization parameters at the current test iteration. Further optionally, the scaling coefficients corresponding to the iterations in the target iteration interval can be consistent. Furthermore, the offsets corresponding to the iterations in the target iteration interval are consistent. At this time, the data bit width, point position and other quantization parameters corresponding to the iterations in the target iteration interval are the same, thereby reducing the amount of calculation and improving quantization efficiency and operational efficiency. The specific implementation process is basically the same as that of the above-mentioned embodiment, and reference can be made to the description above, which will not be repeated here.

[0499] When the point position iteration interval is less than the target iteration interval, the processor may update the data bit width, point position, and other quantization parameters at the test iteration corresponding to the target iteration interval, and may also update the point position and other quantization parameters at the sub-test iteration determined by the point position iteration interval. Since the point position and other quantization parameters can be fine-tuned based on the data to be quantized while the data bit width remains unchanged, the point position and other quantization parameters can also be adjusted within the same target iteration interval to further improve quantization accuracy.

[0500] Specifically, the processor may determine a sub-test iteration based on the current test iteration and the point position iteration interval. The sub-test iteration is used to adjust the point position. The sub-test iteration may be an iteration within the target iteration interval. Furthermore, the processor may adjust the point position corresponding to the iteration within the target iteration interval based on the to-be-quantized data of the sub-test iteration and the data bit width corresponding to the target iteration interval. The method for determining the point position can refer to the above formula and is not further described here.

[0501] For example, the current test iteration is the 100th iteration, the target iteration interval is 6, and the iterations included in the target iteration interval are the 100th iteration to the 105th iteration. The point position iteration interval obtained by the processor is I s1=3, the point position can be adjusted every three iterations starting from the current test iteration. Specifically, the processor can use the 100th iteration as the above-mentioned sub-test iteration, and calculate the point position s1 corresponding to the 100th iteration. The point position s1 is shared and quantized in the 100th, 101st, and 102nd iterations. Afterwards, the processor can adjust the point position according to the point position iteration interval I s1 The 103rd iteration is used as the aforementioned sub-test iteration. The processor can also determine the point position s2 corresponding to the second point position iteration interval based on the data to be quantized corresponding to the 103rd iteration and the data bit width n corresponding to the target iteration interval. The point position s2 can then be shared for quantization in iterations 103 through 105. In the disclosed embodiment, the values of the point position s1 before the update and the point position s2 after the update can be the same or different. Furthermore, the processor can re-determine the next target iteration interval and the corresponding quantization parameters, such as the data bit width and point position, based on the data variation range of the data to be quantized, at iteration 106.

[0502] For another example, the current test iteration is the 100th iteration, the target iteration interval is 6, and the iterations included in the target iteration interval are the 101st iteration to the 106th iteration. s1 =3, then the point position can be adjusted every three iterations starting from the current test iteration. Specifically, the processor can determine that the point position corresponding to the first point position iteration interval is s1 based on the data to be quantized in the current test iteration and the target data bit width n1 corresponding to the current test iteration. Then, the point position s1 is shared for quantization in the 101st, 102nd and 103rd iterations. Afterwards, the processor can adjust the point position according to the point position iteration interval I s1 The 104th iteration is used as the aforementioned sub-test iteration. The processor can also determine the point position s2 corresponding to the second point position iteration interval based on the data to be quantized corresponding to the 104th iteration and the data bit width n1 corresponding to the target iteration interval. The point position s2 can then be shared for quantization in iterations 104 through 106. In the disclosed embodiment, the values of the point position s1 before the update and the point position s2 after the update can be the same or different. Furthermore, the processor can re-determine the next target iteration interval and the corresponding quantization parameters, such as the data bit width and point position, based on the data variation range of the data to be quantized, at the 106th iteration.

[0503] Optionally, the point position iteration interval may be equal to 1, meaning that the point position is updated once for each iteration. Optionally, the point position iteration intervals may be the same or different. For example, at least one point position iteration interval included in the target iteration interval may increase sequentially. This is merely an example to illustrate the implementation of this embodiment and is not intended to limit the present disclosure.

[0504] Optionally, the scaling factors corresponding to the iterations in the target iteration interval may be different. Furthermore, the scaling factor may be updated synchronously with the point position, i.e., the iteration interval corresponding to the scaling factor may be equal to the point position iteration interval. That is, whenever the processor updates the determined point position, it will also update the determined scaling factor accordingly.

[0505] Optionally, the offsets corresponding to the iterations in the target iteration interval may also be inconsistent. Furthermore, the offset can be updated synchronously with the above-mentioned point position, that is, the iteration interval corresponding to the offset can be equal to the above-mentioned point position iteration interval. That is, whenever the processor updates the determined point position, the determined offset will be updated accordingly. Of course, the offset can also be updated asynchronously with the above-mentioned location position or data bit width, which is not specifically limited here. Furthermore, the processor can also determine the minimum and maximum values of all elements of the data to be quantized, and further determine quantization parameters such as point position and scaling factor. For details, please refer to the description above.

[0506] In other optional embodiments, the three quantization parameters of point position, scaling factor, and offset may also be asynchronous, i.e., one or all of the point position iteration interval, scaling factor iteration interval, and offset iteration interval are different. The point position iteration interval and scaling factor iteration interval are both less than or equal to the target iteration interval. The offset iteration interval may be less than the target iteration interval. Since the offset is only related to the distribution of the data to be quantized, in an optional embodiment, the offset may also be completely asynchronous with the target iteration interval, i.e., the offset iteration interval may be greater than the target iteration interval.

[0507] In an optional embodiment, the above method can be used in the training or fine-tuning process of a neural network to adjust the quantization parameters of the operation data involved in the fine-tuning or training process of the neural network, so as to improve the quantization accuracy and efficiency of the operation data involved in the neural network operation process. The operation data can be at least one of neuron data, weight data or gradient data. Figure 8LAs shown, according to the data variation curve of the data to be quantized, in the early stage of training or fine-tuning, the differences between the data to be quantized in each iteration are large, and the data variation of the data to be quantized is relatively drastic. At this time, the value of the target iteration interval can be small, so that the quantization parameters in the target iteration interval can be updated in a timely manner to ensure quantization accuracy. In the middle stage of training or fine-tuning, the data variation of the data to be quantized gradually tends to be gentle. At this time, the value of the target iteration interval can be increased to avoid frequent updates of the quantization parameters, thereby improving quantization efficiency and computational efficiency. In the late stage of training or fine-tuning, the training or fine-tuning of the neural network tends to be stable (that is, when the forward operation result of the neural network approaches the preset reference value, the training or fine-tuning of the neural network tends to be stable). At this time, the value of the target iteration interval can be further increased to further improve quantization efficiency and computational efficiency. Based on the above data variation trend, different methods can be used to determine the target iteration interval at different stages of neural network training or fine-tuning to improve quantization efficiency and computational efficiency while ensuring quantization accuracy.

[0508] Specifically, if Figure 8U FIG. 8 is a flow chart showing a method 800U for adjusting a quantization parameter according to another embodiment of the present disclosure. When the method is used in a training or fine-tuning process of a neural network, the method may further include:

[0509] S710: The processor determines whether the current iteration is greater than a first preset iteration;

[0510] The current iteration refers to the iterative operation currently executed by the processor. Optionally, the first preset iteration may be a hyperparameter, the first preset iteration may be determined based on the data variation curve of the data to be quantized, or the first preset iteration may be user-defined. Optionally, the first preset iteration may be less than the total number of iterations included in a training cycle (epoch), wherein a training cycle refers to the completion of one forward operation and one reverse operation for all the data to be quantized in the data set.

[0511] When the current iteration is less than or equal to the first preset iteration, the processor may perform operation S711, use the first preset iteration interval as the target iteration interval, and adjust the quantization parameter according to the first preset iteration interval.

[0512] Optionally, the processor can read the first preset iteration input by the user and determine the first preset iteration interval based on the correspondence between the first preset iteration and the first preset iteration interval. Optionally, the first preset iteration interval can be a hyperparameter, and the first preset iteration interval can also be a user-defined setting. In this case, the processor can directly read the first preset iteration and the first preset iteration interval input by the user, and update the quantization parameter in the neural network operation based on the first preset iteration interval. In the embodiment of the present disclosure, the processor does not need to determine the target iteration interval based on the data variation amplitude of the data to be quantized.

[0513] For example, if the first preset iteration input by the user is the 100th iteration and the first preset iteration interval is 5, then when the current iteration is less than or equal to the 100th iteration, the quantization parameter can be updated according to the first preset iteration interval. That is, the processor can determine that from the 1st iteration to the 100th iteration of the training or fine-tuning of the neural network, the quantization parameter is updated every 5 iterations. Specifically, the processor can determine the quantization parameters such as the data bit width n1 and the point position s1 corresponding to the 1st iteration, and use the quantization parameters such as the data bit width n1 and the point position s1 to quantize the data to be quantized from the 1st iteration to the 5th iteration, that is, the same quantization parameters can be used from the 1st iteration to the 5th iteration. Thereafter, the processor can determine the quantization parameters such as the data bit width n2 and the point position s2 corresponding to the 6th iteration, and use the quantization parameters such as the data bit width n2 and the point position s2 to quantize the data to be quantized from the 6th iteration to the 10th iteration, that is, the same quantization parameters can be used from the 6th iteration to the 10th iteration. Similarly, the processor can perform the quantization in the above manner until the 100th iteration is completed. The determination method of the quantization parameters such as the data bit width and the point position in each iteration interval can be referred to the above description and will not be repeated here.

[0514] For another example, if the first preset iteration input by the user is the 100th iteration, and the first preset iteration interval is 1, then when the current iteration is less than or equal to the 100th iteration, the quantization parameter can be updated according to the first preset iteration interval. That is, the processor can determine that the quantization parameter is updated for each iteration from the 1st iteration to the 100th iteration of the training or fine-tuning of the neural network. Specifically, the processor can determine the quantization parameters such as the data bit width n1 and the point position s1 corresponding to the 1st iteration, and use the quantization parameters such as the data bit width n1 and the point position s1 to quantize the data to be quantized in the 1st iteration. Afterwards, the processor can determine the quantization parameters such as the data bit width n2 and the point position s2 corresponding to the 2nd iteration, and use the quantization parameters such as the data bit width n2 and the point position s2 to quantize the data to be quantized in the 2nd iteration, ... Similarly, the processor can determine quantization parameters such as the data bit width n100 and the dot position s100 for the 100th iteration, and use the quantization parameters such as the data bit width n100 and the dot position s100 to quantize the data to be quantized for the 100th iteration. The method for determining the quantization parameters such as the data bit width and the dot position in each iteration interval can be found in the above description and will not be repeated here.

[0515] The above only illustrates the synchronous update of data bit width and quantization parameters. In other optional embodiments, in each target iteration interval, the processor can also determine the iteration interval of the point position based on the change amplitude of the point position, and update the quantization parameters such as the point position based on the point position iteration interval.

[0516] Optionally, when the current iteration is greater than the first preset iteration, it may indicate that the training or fine-tuning of the neural network is in an intermediate stage. At this time, the data variation range of the to-be-quantized data of the historical iteration may be obtained, and a target iteration interval may be determined based on the data variation range of the to-be-quantized data. This target iteration interval may be greater than the first preset iteration interval, thereby reducing the number of updates to the quantization parameter and improving quantization efficiency and computational efficiency. Specifically, when the current iteration is greater than the first preset iteration, the processor may perform operation S713, determine the target iteration interval based on the data variation range of the to-be-quantized data, and adjust the quantization parameter based on the target iteration interval.

[0517] Continuing with the above example, if the user inputs the first preset iteration as the 100th iteration and the first preset iteration interval is 1, then when the current iteration is less than or equal to the 100th iteration, the quantization parameter can be updated according to the first preset iteration interval. That is, the processor can determine that the quantization parameter is updated for each iteration from the 1st iteration to the 100th iteration of neural network training or fine-tuning. The specific implementation method can be found in the description above. When the current iteration is greater than the 100th iteration, the processor can determine the data variation amplitude of the data to be quantized based on the data to be quantized in the current iteration and the data to be quantized in the previous historical iteration, and determine the target iteration interval based on the data variation amplitude of the data to be quantized. Specifically, when the current iteration is greater than the 100th iteration, the processor can adaptively adjust the data bit width corresponding to the current iteration, obtain the target data bit width corresponding to the current iteration, and use the target data bit width corresponding to the current iteration as the data bit width for the target iteration interval, wherein the data bit widths corresponding to the iterations in the target iteration interval are consistent. At the same time, the processor can determine the point position corresponding to the current iteration based on the target data bit width corresponding to the current iteration and the data to be quantized, and determine the first error based on the point position corresponding to the current iteration. The processor may also determine a quantization error based on the to-be-quantized data corresponding to the current iteration, and determine a second error based on the quantization error. The processor may then determine a target iteration interval based on the first error and the second error. This target iteration interval may be greater than the first preset iteration interval. Furthermore, the processor may determine quantization parameters such as point positions or scaling factors within the target iteration interval. The specific determination method is described above.

[0518] For example, if the current iteration is the 100th iteration, and the processor determines that the iteration interval of the target iteration interval is 3 based on the data variation of the data to be quantized, the processor can determine that the target iteration interval includes 3 iterations, namely the 100th iteration, the 101st iteration, and the 102nd iteration. The processor can also determine the quantization error based on the data to be quantized of the 100th iteration, and determine the second error and the target data bit width corresponding to the 100th iteration based on the quantization error, and use the target data bit width as the data bit width corresponding to the target iteration interval, wherein the data bit widths corresponding to the 100th iteration, the 101st iteration, and the 102nd iteration are all the target data bit width corresponding to the 100th iteration. The processor can also determine quantization parameters such as the point position and scaling factor corresponding to the 100th iteration based on the data to be quantized of the 100th iteration and the target data bit width corresponding to the 100th iteration. Subsequently, the quantization parameters corresponding to the 100th iteration are used to quantize the 100th iteration, the 101st iteration, and the 102nd iteration.

[0519] Further, if Figure 8VFIG. 8 is a flow chart of a quantization parameter adjustment method 800V according to another embodiment of the present disclosure. The method may further include:

[0520] When the current iteration is greater than the first preset iteration, the processor may further perform operation S712, i.e., the processor may further determine whether the current iteration is greater than a second preset iteration. The second preset iteration is greater than the first preset iteration, and the second preset iteration interval is greater than the first preset iteration interval. Optionally, the second preset iteration may be a hyperparameter, and the second preset iteration may be greater than the total number of iterations of at least one training cycle. Optionally, the second preset iteration may be determined based on a data variation curve of the data to be quantized. Optionally, the second preset iteration may be user-defined.

[0521] When the current iteration is greater than or equal to the second preset iteration, the processor may perform operation S714, use the second preset iteration interval as the target iteration interval, and adjust the parameters in the neural network quantization process according to the second preset iteration interval. When the current iteration is greater than the first preset iteration and the current iteration is less than the second preset iteration, the processor may perform the above-mentioned operation S713, determine the target iteration interval based on the data variation amplitude of the to-be-quantized data, and adjust the quantization parameters according to the target iteration interval.

[0522] Optionally, the processor can read a second preset iteration set by the user, and determine a second preset iteration interval based on the correspondence between the second preset iteration and the second preset iteration interval, where the second preset iteration interval is greater than the first preset iteration interval. Optionally, when the convergence degree of the neural network meets a preset condition, it is determined that the current iteration is greater than or equal to the second preset iteration. For example, when the forward operation result of the current iteration approaches a preset reference value, it can be determined that the convergence degree of the neural network meets the preset condition, and at this time, it can be determined that the current iteration is greater than or equal to the second preset iteration. Alternatively, when the loss value corresponding to the current iteration is less than or equal to a preset threshold, it can be determined that the convergence degree of the neural network meets the preset condition.

[0523] Optionally, the second preset iteration interval may be a hyperparameter, and the second preset iteration interval may be greater than or equal to the total number of iterations of at least one training cycle. Optionally, the second preset iteration interval may be user-defined. The processor may directly read the second preset iteration and the second preset iteration interval input by the user, and update the quantization parameter in the neural network operation according to the second preset iteration interval. For example, the second preset iteration interval may be equal to the total number of iterations of a training cycle, that is, the quantization parameter is updated once per training cycle (epoch).

[0524] Furthermore, the above method also includes:

[0525] When the current iteration is greater than or equal to the second preset iteration, the processor may further determine whether the current data bit width needs to be adjusted at each verification iteration. If the current data bit width needs to be adjusted, the processor may switch from the above-mentioned operation S714 to operation S713 to re-determine the data bit width so that the data bit width can meet the requirements of the data to be quantized.

[0526] Specifically, the processor can determine whether the data bit width needs to be adjusted based on the above-mentioned second error. The processor can also perform the above-mentioned operation S715 to determine whether the second error is greater than a preset error value. When the current iteration is greater than or equal to the second preset iteration and the second error is greater than the preset error value, the processor switches to perform operation S713 to determine the iteration interval based on the data variation range of the data to be quantized, so as to redetermine the data bit width based on the iteration interval. If the current iteration is greater than or equal to the second preset iteration and the second error is less than or equal to the preset error value, the processor continues to perform operation S714, uses the second preset iteration interval as the target iteration interval, and adjusts the parameters in the neural network quantization process based on the second preset iteration interval. The preset error value can be determined based on a preset threshold corresponding to the quantization error. When the second error is greater than the preset error value, it indicates that the data bit width may need to be further adjusted. The processor can determine the iteration interval based on the data variation range of the data to be quantized, so as to redetermine the data bit width based on the iteration interval.

[0527] For example, the second preset iteration interval is the total number of iterations in a training cycle. When the current iteration is greater than or equal to the second preset iteration, the processor may update the quantization parameter according to the second preset iteration interval, that is, update the quantization parameter once per training cycle (epoch). In this case, the starting iteration of each training cycle serves as a test iteration. At the starting iteration of each training cycle, the processor may determine the quantization error based on the data to be quantized in the test iteration, determine the second error based on the quantization error, and determine whether the second error is greater than the preset error value according to the following formula:

[0528] diff update2 =θ*diff bit 2 >T

[0529] Among them, diff update2 Indicates the second error, diff bit represents the quantization error, θ represents a hyperparameter, and T represents a preset error value. Optionally, the preset error value may be equal to the first preset threshold divided by the hyperparameter. Of course, the preset error value may also be a hyperparameter. For example, the preset error value may be calculated according to the following formula: T = th / 10, where th represents the first preset threshold and the hyperparameter is 10.

[0530] If the second error diff update2 If the error is greater than the preset error value T, it means that the data bit width may not meet the preset requirements. In this case, the second preset iteration interval can no longer be used to update the quantization parameter. The processor can determine the target iteration interval according to the data variation range of the data to be quantized to ensure that the data bit width meets the preset requirements. update2 When the error is greater than the predetermined error value T, the processor switches from the above-mentioned operation S714 to the above-mentioned operation S713.

[0531] Of course, in other embodiments, the processor can determine whether the data bit width needs to be adjusted based on the above-mentioned quantization error. For example, the second preset iteration interval is the total number of iterations of a training cycle. When the current iteration is greater than or equal to the second preset iteration, the processor can update the quantization parameter according to the second preset iteration interval, that is, update the quantization parameter once per training cycle (epoch). Among them, the starting iteration of each training cycle serves as a test iteration. At the starting iteration of each training cycle, the processor can determine the quantization error based on the data to be quantized of the test iteration, and when the quantization error is greater than or equal to the first preset threshold, it means that the data bit width may not meet the preset requirements, that is, the processor switches from the above-mentioned operation S714 to the above-mentioned operation S713.

[0532] In an optional embodiment, the aforementioned quantization parameters such as point position, scaling factor, and offset can be displayed via a display device. In this case, the user can obtain the quantization parameters during the neural network operation via the display device, and the user can also adaptively modify the quantization parameters determined by the processor. Similarly, the aforementioned data bit width and target iteration interval can also be displayed via a display device. In this case, the user can obtain the parameters such as the target iteration interval and data bit width during the neural network operation via the display device, and the user can also adaptively modify the parameters such as the target iteration interval and data bit width determined by the processor.

[0533] It should be noted that the above-mentioned target iteration intervals for determining the data bit width and the target iteration intervals for the quantization parameters are only examples of some cases, not exhaustive. Those skilled in the art, while understanding the essence of the disclosed technical solution, may produce other deformations or transformations based on the disclosed technical solution. For example, determining the target iteration interval for the quantization parameters within the target iteration interval for determining the data bit width is also applicable to Figure 6 、 Figure 7 and Figure 8A However, as long as the functions and technical effects achieved are similar to those of this disclosure, they should fall within the scope of protection of this disclosure.

[0534] This technical solution is used to determine the quantization parameter, adjust the data bit width or quantization parameter according to the quantization error, and determine the target iteration interval for adjusting the data bit width or quantization parameter, so as to adjust the data bit width or quantization parameter at an appropriate time point during the neural network operation, so that the appropriate quantization parameter is used at the appropriate iteration time point, and the artificial intelligence processor chip can execute the neural network operation at the speed of fixed-point operation, thereby improving the peak computing power of the artificial intelligence processor chip while meeting the accuracy of floating-point operations required for the operation.

[0535] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this disclosure is not limited by the order of the actions described, because according to this disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required for this disclosure.

[0536] It should be further explained that although Figure 2 、 Figure 6 、 Figure 7 、 Figures 8A to 8V The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 、 Figure 6 、 Figure 7 、 Figures 8A to 8V At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0537] like Figure 9 As shown in FIG, it is a block diagram of the hardware configuration of a device for determining a quantization parameter of a neural network proposed in the present disclosure. Figure 9 In the embodiment, the quantization parameter determination device 10 of the neural network may include a processor 110 and a memory 120. Figure 9 In the apparatus 10 for determining the quantization parameter of a neural network, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the apparatus 10 for determining the quantization parameter of a neural network may also include Figure 10 The components shown in the figure are different from the common components. For example: fixed-point arithmetic units.

[0538] The apparatus 10 for determining a quantization parameter of a neural network may correspond to a computing device having various processing functions, such as functions for generating a neural network, training or learning a neural network, quantizing a floating-point neural network to a fixed-point neural network, or retraining a neural network. For example, the apparatus 10 for determining a quantization parameter of a neural network may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0539] The processor 110 controls all functions of the apparatus 10 for determining a quantization parameter of a neural network. For example, the processor 110 controls all functions of the apparatus 10 for determining a quantization parameter of a neural network by executing a program stored in the memory 120 on the apparatus 10 for determining a quantization parameter of a neural network. The processor 110 may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the apparatus 10 for determining a quantization parameter of a neural network. However, the present disclosure is not limited thereto.

[0540] Memory 120 is hardware used to store various data processed by the apparatus 10 for determining quantization parameters of a neural network. For example, memory 120 can store processed data and data to be processed by the apparatus 10 for determining quantization parameters of a neural network. Memory 120 can store data sets involved in the neural network operations that have been processed or are to be processed by the processor 110, such as data from an untrained initial neural network, intermediate data from a neural network generated during training, data from a neural network that has completed all training, data from a quantized neural network, and so on. Furthermore, memory 120 can store applications, driver programs, and the like to be driven by the apparatus 10 for determining quantization parameters of a neural network. For example, memory 120 can store various programs related to the neural network training algorithm, quantization algorithm, and the like to be executed by the processor 110. Memory 120 can be DRAM, but the present disclosure is not limited thereto. Memory 120 can include at least one of volatile memory and non-volatile memory. The non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous DRAM (SDRAM), a PRAM, an MRAM, an RRAM, a ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 120 may include at least one of a hard disk drive (HDD), a solid-state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, caches, or a memory stick.

[0541] Processor 110 can generate a trained neural network by repeatedly training (learning) a given initial neural network. In this state, to ensure the processing accuracy of the neural network, the parameters of the initial neural network are in a high-precision data representation format, such as a data representation format with 32-bit floating-point precision. The parameters can include various types of data input to / output from the neural network, such as input / output neurons, weights, biases, etc. Compared to fixed-point arithmetic, floating-point arithmetic requires a relatively large number of operations and relatively frequent memory accesses. Specifically, the majority of operations required for neural network processing are known to be various convolution operations. Therefore, in mobile devices with relatively low processing power (such as smartphones, tablets, wearable devices, embedded devices, etc.), high-precision neural network data operations can lead to underutilization of mobile device resources. Consequently, in order to drive neural network operations within an acceptable precision loss range and minimize the amount of computation in such devices, the high-precision data involved in the neural network operations can be quantized and converted into low-precision fixed-point numbers.

[0542] Taking into account the processing performance of devices such as mobile devices and embedded devices that deploy neural networks, the neural network quantization parameter determination device 10 performs quantization to convert the parameters of the trained neural network into fixed-point quantization with a specific number of bits, and the neural network quantization parameter determination device 10 sends the corresponding quantization parameters to the device that deploys the neural network, so that when the artificial intelligence processor chip performs training, fine-tuning, and other operations, fixed-point operations are performed. The device that deploys the neural network can be an autonomous vehicle, a robot, a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, etc. that performs speech recognition, image recognition, etc. by using the neural network, but the present disclosure is not limited thereto.

[0543] The processor 110 obtains data from the memory 120 during the neural network operation process. The data includes at least one of neurons, weights, biases and gradients. Figure 2 The technical solution shown determines corresponding quantization parameters and uses them to quantize the target data during the neural network operation. The neural network operation is then performed on the quantized data. This operation includes, but is not limited to, training, fine-tuning, and inference.

[0544] The processor 110 calculates the quantization error diff bit The data bit width n is adjusted, and the processor 110 can execute Figure 6 、 Figure 7 The procedure of the method for the target iteration interval shown in FIG8 is used to determine the target iteration interval of the data bit width or the target iteration interval of the quantization parameter.

[0545] In summary, the specific functions implemented by the memory 120 and the processor 110 of a device for determining quantization parameters of a neural network provided in the embodiments of this specification can be explained in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, and will not be repeated here.

[0546] In this embodiment, the processor 110 may be implemented in any suitable manner. For example, the processor 110 may take the form of a microprocessor or a processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, an embedded microcontroller, and the like.

[0547] like Figure 10 FIG2 is a schematic diagram showing the application of the device for determining the quantization parameters of a neural network proposed in the present disclosure to an artificial intelligence processor chip. Figure 10 As described above, in the quantization parameter determination device 10 of a neural network such as a PC or a server, the processor 110 performs a quantization operation to quantize the floating-point data involved in the neural network operation process into fixed-point numbers, and the fixed-point operator on the artificial intelligence processor chip uses the fixed-point numbers obtained by quantization to perform training, fine-tuning or inference. The artificial intelligence processor chip is a dedicated hardware for driving neural networks. Since the artificial intelligence processor chip is implemented with relatively low power or performance, the present technical solution uses low-precision fixed-point numbers to implement neural network operations. Compared with high-precision data, the memory bandwidth required to read low-precision fixed-point numbers is smaller, and the caches of the artificial intelligence processor chip can be better used to avoid memory access bottlenecks. At the same time, when executing SIMD instructions on the artificial intelligence processor chip, more calculations are achieved within one clock cycle, achieving faster execution of neural network operations.

[0548] Furthermore, when comparing fixed-point and high-precision data operations of the same length, especially when comparing fixed-point and floating-point operations, it is clear that floating-point operations have a more complex calculation model and require more logic devices to construct a floating-point unit. Therefore, floating-point units are physically larger than fixed-point units. Furthermore, floating-point units consume more processing resources, resulting in a power consumption difference of orders of magnitude between fixed-point and floating-point operations.

[0549] In summary, this technical solution enables the replacement of floating-point units on AI processor chips with fixed-point units, resulting in lower power consumption. This is particularly important for mobile devices. This opens the door to a wide range of embedded systems that cannot efficiently run floating-point code, enabling widespread adoption in the IoT.

[0550] In this technical solution, the artificial intelligence processor chip may correspond to, for example, a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, etc., which are dedicated chips for driving neural networks, but this disclosure is not limited to this.

[0551] In this technical solution, the artificial intelligence processor chip can be implemented in a separate device independent of the neural network quantization parameter determination device 10, or the neural network quantization parameter determination device 10 can also be implemented as a functional module of the artificial intelligence processor chip. However, this disclosure is not limited to this.

[0552] In this technical solution, the operating system of a general-purpose processor (such as a CPU) generates instructions based on this technical solution, and sends the generated instructions to an artificial intelligence processor chip (such as a GPU), which executes the instruction operations to realize the determination of the quantization parameters of the neural network and the quantization process. There is another application scenario, in which the general-purpose processor directly determines the corresponding quantization parameters based on this technical solution, and the general-purpose processor directly quantizes the corresponding target data according to the quantization parameters, and the artificial intelligence processor chip uses the quantized data to perform fixed-point operations. What's more, the general-purpose processor (such as a CPU) and the artificial intelligence processor chip (such as a GPU) operate in a pipelined manner, and the operating system of the general-purpose processor (such as a CPU) generates instructions based on this technical solution, and while copying the target data, the artificial intelligence processor chip (such as a GPU) performs neural network operations, so that certain time consumption can be hidden. However, the present disclosure is not limited to this.

[0553] In this embodiment, the disclosed embodiment further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed, the method for determining the quantization parameters of the neural network described above is implemented.

[0554] As can be seen from the above, during the neural network operation process, the technical solution disclosed herein is used to determine the quantization parameters during quantization. The quantization parameters are used by the artificial intelligence processor to quantize the data in the neural network operation process, converting high-precision data into low-precision fixed-point numbers, which can reduce the size of all data storage spaces involved in the neural network operation process. For example, converting float32 to fix8 can reduce the model parameters by 4 times. Since the data storage space becomes smaller, the neural network uses a smaller space when deployed, so that the on-chip memory on the artificial intelligence processor chip can accommodate more data, reducing the memory access data of the artificial intelligence processor chip and improving computing performance.

[0555] Those skilled in the art will also appreciate that, in addition to implementing the client and server in pure computer-readable program code, it is entirely possible to implement the same functionality by programming the method steps through logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Therefore, such a client and server can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both software modules implementing the method and structures within the hardware component.

[0556] like Figure 11 FIG. 1 is a functional block diagram of a device for determining quantization parameters of a neural network proposed in the present disclosure. The method includes:

[0557] A statistical unit a is used to count the data to be quantified and determine the statistical results of each type of data to be quantified; wherein the data to be quantified includes at least one of the neurons, weights, gradients, and biases of the neural network;

[0558] The quantization parameter determination unit b is used to determine the corresponding quantization parameter using the statistical results of each type of data to be quantized and the data bit width; wherein the quantization parameter is used by the artificial intelligence processor to perform corresponding quantization on the data in the neural network operation process.

[0559] In this embodiment, optionally, the device for determining a quantization parameter of a neural network further includes:

[0560] The first quantization unit is configured to quantize the data to be quantized using a corresponding quantization parameter.

[0561] In this embodiment, optionally, the device for determining a quantization parameter of a neural network further includes:

[0562] The second quantization unit is configured to quantize the target data using a corresponding quantization parameter; wherein the characteristics of the target data and the characteristics of the data to be quantized are similar.

[0563] In this embodiment, the neural network operation process includes at least one operation of neural network training, neural network reasoning, and neural network fine-tuning.

[0564] In this embodiment, the statistical results obtained by the statistical unit are the maximum value and the minimum value of each type of data to be quantified.

[0565] In this embodiment, the statistical result obtained by the statistical unit is the maximum absolute value of each type of data to be quantified.

[0566] In this embodiment, the statistical unit determines the absolute maximum value according to the maximum value and the minimum value of each type of data to be quantified.

[0567] In this embodiment, the quantization parameter determination unit determines the quantization parameter according to the maximum value, the minimum value and the data bit width of each type of data to be quantized.

[0568] In this embodiment, the quantization parameter determination unit determines the quantization parameter according to the maximum absolute value of each type of data to be quantized and the data bit width.

[0569] In this embodiment, the quantization parameter determined by the quantization parameter determination unit is a point position parameter or a first scaling coefficient.

[0570] In this embodiment, the quantization parameter determination unit determines the first scaling coefficient based on the point position parameter and the second scaling coefficient; wherein, the point position parameter used when determining the first scaling coefficient is a known fixed value, or the result of multiplying the point position parameter and the corresponding second scaling coefficient is used as the first scaling coefficient as a whole for data quantization in the neural network operation process.

[0571] In this embodiment, the quantization parameter determined by the quantization parameter determination unit includes a point position parameter and a second scaling coefficient.

[0572] In this embodiment, the quantization parameter determination unit determines the second scaling factor according to the point position parameter, the statistical result, and the data bit width.

[0573] In this embodiment, the quantization parameter determined by the quantization parameter determination unit further includes an offset.

[0574] In this embodiment, the quantization parameter determination unit determines the offset according to a statistical result of each type of data to be quantized.

[0575] In this embodiment, the data bit width used by the quantization parameter determination unit is a preset value.

[0576] In this embodiment, the quantization parameter determination unit includes an adjustment module and a quantization error determination module; wherein,

[0577] The adjustment module is used to adjust the data bit width according to the corresponding quantization error;

[0578] The quantization error determination module is used to determine the quantization error according to the quantized data and the corresponding pre-quantization data.

[0579] In this embodiment, the adjustment module is specifically configured to:

[0580] The quantization error is compared with a threshold, and the data bit width is adjusted according to the comparison result; wherein the threshold includes at least one of a first threshold and a second threshold.

[0581] In this embodiment, the adjustment module includes a first adjustment submodule, wherein the first adjustment submodule is configured to:

[0582] If the quantization error is greater than or equal to the first threshold, the data bit width is increased.

[0583] In this embodiment, the adjustment module includes a second adjustment submodule, wherein the second adjustment submodule is configured to:

[0584] If the quantization error is less than or equal to the second threshold, the data bit width is reduced.

[0585] In this embodiment, the adjustment module includes a third adjustment submodule, wherein the third adjustment submodule is configured to:

[0586] If the quantization error is between the first threshold and the second threshold, the data bit width remains unchanged.

[0587] In this embodiment, the quantization error determination module includes:

[0588] a quantization interval determination submodule, configured to determine the quantization interval according to the data bit width;

[0589] The first quantization error determination submodule is configured to determine the quantization error according to the quantization interval, the number of the quantized data, and the corresponding pre-quantization data.

[0590] In this embodiment, the quantization error determination module includes:

[0591] a dequantized data determination submodule, configured to dequantize the quantized data to obtain dequantized data; wherein the data format of the dequantized data is the same as the data format of the corresponding pre-quantized data;

[0592] The second quantization error determination submodule is configured to determine a quantization error based on the quantized data and the corresponding inverse quantization data.

[0593] In this embodiment, the data before quantization used by the quantization error determination module is the data to be quantized.

[0594] In this embodiment, the pre-quantization data used by the quantization error determination module is the data to be quantized involved in the weight update iteration process within the target iteration interval; wherein, the target iteration interval includes at least one weight update iteration, and the same data bit width is used in the quantization process within the same target iteration interval.

[0595] In this embodiment, the neural network quantization parameter determination device further includes a first target iteration interval determination unit; wherein the first target iteration interval determination unit includes:

[0596] A first change trend value determination module is configured to determine, at a pre-determined time point, a change trend value of a point position parameter of the data to be quantized involved in the weight update iteration process; wherein the pre-determined time point is a time point for determining whether the data bit width needs to be adjusted, and the pre-determined time point corresponds to a time point when the weight update iteration is completed;

[0597] The first target iteration interval module is used to determine the corresponding target iteration interval according to the change trend value of the point position parameter.

[0598] In this embodiment, the first target iteration interval determining unit includes:

[0599] A second change trend value determination module is configured to determine, at a pre-determined time point, a change trend value of a point position parameter of the to-be-quantized data and a change trend value of a data bit width involved in the weight update iteration process; wherein the pre-determined time point is a time point for determining whether the data bit width needs to be adjusted, and the pre-determined time point corresponds to a time point when the weight update iteration is completed;

[0600] The second target iteration interval module is configured to determine the corresponding target iteration interval according to the change trend value of the point position parameter and the change trend value of the data bit width.

[0601] In this embodiment, the first target iteration interval determining unit further includes a first predicted time point determining unit; wherein,

[0602] The first predicted time point determining unit is configured to determine the first predicted time point according to the target iteration interval.

[0603] In this embodiment, the first target iteration interval determination unit also includes a second prediction time point determination unit; wherein, the second prediction time point determination unit is used to determine the second prediction time point based on the data change amplitude curve; wherein, the data change amplitude curve is obtained by statistically analyzing the data change amplitude during the weight update iteration process.

[0604] In this embodiment, the first change trend value determination module and the second change trend value determination module both determine the change trend value of the point position parameter based on the sliding average of the point position parameter corresponding to the current prediction time point and the sliding average of the point position parameter corresponding to the previous prediction time point.

[0605] In this embodiment, the first change trend value determination module and the second change trend value determination module both determine the change trend value of the point position parameter based on the sliding average of the point position parameter corresponding to the current predicted time point and the point position parameter corresponding to the previous predicted time point.

[0606] In this embodiment, the first change trend value determining module and the second change trend value determining module both include:

[0607] A point position parameter determination submodule corresponding to the current predicted time point is used to determine the point position parameter corresponding to the current predicted time point based on the point position parameter corresponding to the previous predicted time point and the adjustment value of the data bit width;

[0608] an adjustment result determination submodule, configured to adjust the sliding average of the point position parameter corresponding to the last predicted time point according to the adjustment value of the data bit width to obtain an adjustment result;

[0609] The first sliding average value determination submodule is used to determine the sliding average value of the point position parameter corresponding to the current predicted time point according to the point position parameter corresponding to the current predicted time point and the adjustment result.

[0610] In this embodiment, the first change trend value determining module and the second change trend value determining module both include:

[0611] An intermediate result determination submodule, configured to determine an intermediate result of a sliding average of the point position parameters corresponding to the current predicted time point based on the point position parameters corresponding to the previous predicted time point and the sliding average of the point position parameters corresponding to the previous predicted time point;

[0612] The second sliding average value determination submodule is used to determine the sliding average value of the point position parameter corresponding to the current predicted time point according to the intermediate result of the sliding average value of the point position parameter corresponding to the current predicted time point and the adjustment value of the data bit width.

[0613] In this embodiment, the second change trend value determining module determines the change trend value of the data bit width according to the corresponding quantization error.

[0614] In this embodiment, the first target iteration interval determining unit further includes:

[0615] A quantization error determination module is used to determine a corresponding quantization error; wherein the pre-quantization data corresponding to the quantization error is the data to be quantized involved in the weight update iteration process corresponding to the predicted time point;

[0616] The data bit width determination module is used to determine the data bit width used in the quantization process within the target iteration interval according to the corresponding quantization error.

[0617] In this embodiment, the data bit width determination module is specifically configured to:

[0618] The quantization error is compared with a threshold, and based on the comparison result, the data bit width used in the quantization process in the previous target iteration interval is adjusted, and the adjustment result is used as the data bit width used in the quantization process in the current target iteration interval.

[0619] In this embodiment, the pre-quantization data used by the quantization error determination module is the data to be quantized involved in the weight update iteration within the target iteration interval; wherein, the target iteration interval includes at least one weight update iteration, and the same quantization parameters are used in the quantization process within the same target iteration interval.

[0620] In this embodiment, the neural network quantization parameter determination device further includes a second target iteration interval determination unit; wherein the second target iteration interval determination unit includes:

[0621] a third change trend value determination module, configured to determine, at a predicted time point, a change trend value of a point position parameter of the to-be-quantized data involved in the weight update iteration process; wherein the predicted time point is a time point for determining whether the quantization parameter needs to be adjusted, and the predicted time point corresponds to a time point when the weight update iteration is completed;

[0622] The third target iteration interval module is used to determine the corresponding target iteration interval according to the change trend value of the point position parameter.

[0623] In this embodiment, the quantization parameter determination unit determines the point position parameter according to the statistical result and the data bit width.

[0624] In one embodiment, a device for adjusting data bit width is provided, comprising:

[0625] An acquisition module, configured to acquire a data bit width for performing a quantization process on the data to be quantized, wherein the data bit width indicates a bit width of quantized data of the data to be quantized after the quantization process;

[0626] a quantization module, configured to perform quantization processing on a set of data to be quantized based on the data bit width, so as to convert the set of data to be quantized into a set of quantized data, wherein the set of quantized data has the data bit width;

[0627] a determination module, configured to compare the set of data to be quantized with the set of quantized data to determine a quantization error associated with the data bit width; and

[0628] The data bit width is adjusted based on the determined quantization error.

[0629] In this embodiment, the comparison module includes:

[0630] an interval determination module, configured to determine a quantization interval according to the data bit width; and

[0631] The error determination module is configured to determine the quantization error according to the quantization interval, the set of quantized data, and the set of data to be quantized.

[0632] In this embodiment, the error determination module includes:

[0633] a quantization module, configured to perform inverse quantization on the set of quantized data according to the quantization interval to obtain a set of inverse quantized data, wherein a data format of the set of inverse quantized data is the same as a data format of the set of data to be quantized; and

[0634] The quantization error determination module is used to determine the quantization error according to the set of inverse quantized data and the set of data to be quantized.

[0635] In this embodiment, the adjustment module includes:

[0636] a comparing module, configured to compare the quantization error with a preset threshold, wherein the preset threshold comprises at least one of a first threshold and a second threshold; and

[0637] The bit width adjustment module is configured to adjust the data bit width according to the comparison result.

[0638] In this embodiment, the bit width adjustment module includes:

[0639] An increasing module is configured to increase the data bit width in response to determining that the quantization error is greater than or equal to a first threshold.

[0640] In this embodiment, the adding module includes:

[0641] The step-length increasing module is configured to increase the data bit width according to a first preset bit width step to determine an adjusted data bit width.

[0642] In this embodiment, the quantization module is further configured to perform quantization processing on the set of data to be quantized based on the adjusted data bit width, so as to convert the set of data to be quantized into another set of quantized data, wherein the another set of quantized data has the adjusted data bit width; and

[0643] The determination module is further configured to compare the set of data to be quantized with the other set of quantized data, and determine another quantization error associated with the adjusted data bit width, until the another quantization error is smaller than the first preset threshold.

[0644] In this embodiment, the device is called iteratively.

[0645] In this embodiment, the adjustment module includes:

[0646] A reducing module is configured to reduce the data bit width in response to determining that the quantization error is less than or equal to the second threshold.

[0647] In this embodiment, the reduction module includes:

[0648] The step size reduction module is configured to reduce the data bit width according to a second preset bit width step size to determine an adjusted data bit width.

[0649] In this embodiment, the quantization module is further configured to perform quantization processing on the set of data to be quantized based on the adjusted data bit width, so as to convert the set of data to be quantized into another set of quantized data, wherein the another set of quantized data has the adjusted data bit width; and

[0650] The determining module is further configured to determine another quantization error associated with the adjusted data bit width based on the set of data to be quantized and the another set of quantized data, until the another quantization error is greater than the second preset threshold.

[0651] In this embodiment, the device is called iteratively.

[0652] In this embodiment, the adjustment module includes:

[0653] A maintaining module is configured to maintain the data bit width in response to determining that the quantization error is between the first threshold and the second threshold.

[0654] In this embodiment, it further includes:

[0655] an updating module, configured to update a quantization parameter used for performing quantization processing on the set of data to be quantized based on the set of data to be quantized and the adjusted data bit width; and

[0656] The quantization module is further configured to perform quantization processing on the set of data to be quantized based on the updated quantization parameter.

[0657] In this embodiment, the amplitude module is used to obtain the data variation amplitude of the data to be quantified;

[0658] The interval module is configured to determine a target iteration interval according to a data variation range of the data to be quantized, so as to adjust the data bit width according to the target iteration interval, wherein the target iteration interval includes at least one iteration.

[0659] In this embodiment, the amplitude module includes:

[0660] The point position module is used to obtain the variation range of the point position, wherein the variation range of the point position is used to represent the data variation range of the data to be quantified, and the variation range of the point position is positively correlated with the data variation range of the data to be quantified.

[0661] In this embodiment, the point location module includes:

[0662] A first mean module is configured to determine a first mean based on a point position corresponding to a previous test iteration before a current test iteration and a point position corresponding to a historical iteration before the previous test iteration, wherein the previous test iteration is a test iteration corresponding to a previous iteration interval before the target iteration interval;

[0663] a second mean value module, configured to determine a second mean value based on a point position corresponding to the current test iteration and a point position of a historical iteration before the current test iteration; wherein the point position corresponding to the current test iteration is determined based on a target data bit width corresponding to the current test iteration and the data to be quantized;

[0664] A first error module is configured to determine a first error based on the first mean and the second mean, where the first error is used to characterize a variation range of the point position.

[0665] In this embodiment, the spacing module includes:

[0666] A first interval module is configured to determine the target iteration interval according to the first error, where the target iteration interval is negatively correlated with the first error.

[0667] In this embodiment, the amplitude module further includes:

[0668] A trend module, used to obtain the change trend of the data bit width;

[0669] The data variation amplitude module is used to determine the data variation amplitude of the data to be quantized according to the variation amplitude of the point position and the variation trend of the data bit width.

[0670] In this embodiment, the spacing module further includes:

[0671] The iteration interval module is used to determine the target iteration interval based on the obtained first error and second error; wherein the first error is used to characterize the variation range of the point position, and the second error is used to characterize the variation trend of the data bit width.

[0672] In this embodiment, the iteration interval module includes:

[0673] a target error module, configured to take the maximum value of the first error and the second error as a target error;

[0674] A target iteration interval module is configured to determine the target iteration interval according to the target error, wherein the target error is negatively correlated with the target iteration interval.

[0675] In this embodiment, the second error is determined according to a quantization error;

[0676] The quantization error is determined according to the data to be quantized in the current test iteration and the quantized data of the current test iteration, and the second error is positively correlated with the quantization error.

[0677] In this embodiment, the apparatus is used for training or fine-tuning a neural network, and the apparatus further comprises:

[0678] The first iteration module is configured to determine a target iteration interval according to a data variation range of the data to be quantized when the current iteration is greater than a first preset iteration, and adjust the quantization parameter according to the target iteration interval.

[0679] In this embodiment, the device further comprises:

[0680] The second iteration module is configured to use the first preset iteration interval as the target iteration interval when the current iteration is less than or equal to the first preset iteration interval, and adjust the quantization parameter according to the first preset iteration interval.

[0681] In this embodiment, the device further comprises:

[0682] a third iteration module, configured to, when the current iteration is greater than or equal to a second preset iteration, use the second preset iteration interval as the target iteration interval, and adjust the quantization parameter according to the second preset iteration interval;

[0683] The second preset iteration is greater than the first preset iteration, and the second preset iteration interval is greater than the first preset iteration interval.

[0684] In this embodiment, the device further comprises:

[0685] The convergence module is configured to determine that the current test iteration is greater than or equal to a second preset iteration when the convergence degree of the neural network meets a preset condition.

[0686] In this embodiment, the device further comprises:

[0687] When the current iteration is greater than or equal to a second preset iteration and the second error is greater than a preset error value, an iteration interval is determined according to a data variation amplitude of the data to be quantized, so as to redetermine the data bit width according to the iteration interval.

[0688] In one embodiment, a computer-rea...

Claims

1. A method for adjusting the data bit width in a convolutional neural network layer during a neural network operation, characterized in that: The method comprises: Acquire a data bit width for performing quantization processing on data to be quantized, the data to be quantized including at least one of neurons, weights, gradients, and biases, the data bit width indicating a data bit width of quantized data of the data to be quantized after the quantization processing; Based on the data bit width, performing quantization processing on a set of data to be quantized to convert the set of data to be quantized into a set of quantized data, wherein the set of quantized data has the data bit width; comparing the set of data to be quantized and the set of quantized data to determine a quantization error associated with the data bit width; adjusting the data bit width based on the determined quantization error; and Applying the adjusted data bit width during quantization in the convolutional neural network layer; Wherein adjusting the data bit width based on the determined quantization error comprises: comparing the quantization error with a preset threshold, the preset threshold comprising at least one of a first threshold and a second threshold; and adjusting the data bit width according to a result of the comparison; Wherein adjusting the data bit width according to the comparison result includes: In response to determining that the quantization error is greater than or equal to a first threshold, increasing the data bit width; The neural network is used for one or more of the following: image recognition or speech recognition.

2. The method according to claim 1, characterized in that Comparing the set of data to be quantized with the set of quantized data to determine a quantization error associated with the data bit width includes: Determining a quantization interval according to the data bit width; and The quantization error is determined according to the quantization interval, the set of quantized data, and the set of data to be quantized.

3. The method according to claim 2, characterized in that Determining the quantization error according to the quantization interval, the set of quantized data, and the set of data to be quantized includes: performing inverse quantization on the set of quantized data according to the quantization interval to obtain a set of inverse quantized data, wherein a data format of the set of inverse quantized data is the same as a data format of the set of data to be quantized; and A quantization error is determined according to the set of inverse quantized data and the set of data to be quantized.

4. The method according to claim 1, wherein Increasing the data bit width includes: The data bit width is increased according to a first preset bit width step to determine an adjusted data bit width.

5. The method according to claim 4, characterized in that The method further comprises: Based on the adjusted data bit width, performing a quantization process on the set of data to be quantized to convert the set of data to be quantized into another set of quantized data, the another set of quantized data having the adjusted data bit width; and The set of data to be quantized is compared with the other set of quantized data to determine another quantization error associated with the adjusted data bit width, until the other quantization error is less than the first preset threshold.

6. The method according to claim 5, characterized in that The method is performed iteratively.

7. The method according to claim 1, characterized in that Adjusting the data bit width according to the comparison result further includes: In response to determining that the quantization error is less than or equal to the second threshold, the data bit width is reduced.

8. The method according to claim 7, characterized in that Reducing the data bit width includes: The data bit width is reduced according to a second preset bit width step to determine an adjusted data bit width.

9. The method according to claim 8, characterized in that The method further comprises: Based on the adjusted data bit width, performing a quantization process on the set of data to be quantized to convert the set of data to be quantized into another set of quantized data, the another set of quantized data having the adjusted data bit width; and Based on the set of data to be quantized and the another set of quantized data, another quantization error associated with the adjusted data bit width is determined until the another quantization error is greater than the second preset threshold.

10. The method according to claim 9, characterized in that The method is performed iteratively.

11. The method according to claim 1, wherein Adjusting the data bit width according to the comparison result further includes: In response to determining that the quantization error is between the first threshold and the second threshold, the data bit width is maintained.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: updating a quantization parameter for performing a quantization process on the set of data to be quantized based on the set of data to be quantized and the adjusted data bit width; and Based on the updated quantization parameter, a quantization process is performed on the set of data to be quantized.

13. The method according to claim 1, wherein The method further comprises: Obtaining the data variation range of the data to be quantified; and A target iteration interval is determined according to a data variation range of the data to be quantized, so as to adjust the data bit width according to the target iteration interval, wherein the target iteration interval includes at least one iteration.

14. The method according to claim 13, characterized in that The obtaining of the data variation range of the data to be quantified includes: The variation range of the point position is obtained, wherein the variation range of the point position is used to represent the data variation range of the data to be quantified, and the variation range of the point position is positively correlated with the data variation range of the data to be quantified.

15. The method according to claim 14, characterized in that The variation range of the acquisition point position includes: Determining a first mean value based on a point position corresponding to a previous test iteration before the current test iteration and a point position corresponding to a historical iteration before the previous test iteration, wherein the previous test iteration is a test iteration corresponding to a previous iteration interval before the target iteration interval; Determining a second mean value based on a point position corresponding to the current test iteration and a point position of a historical iteration before the current test iteration, wherein the point position corresponding to the current test iteration is determined based on a target data bit width corresponding to the current test iteration and the data to be quantized; A first error is determined according to the first mean and the second mean, where the first error is used to characterize a variation range of the point position.

16. The method according to claim 15, characterized in that The step of determining a target iteration interval according to a data variation range of the data to be quantified includes: The target iteration interval is determined according to the first error, and the target iteration interval is negatively correlated with the first error.

17. The method according to claim 14, characterized in that The obtaining of the data variation range of the data to be quantified further includes: Obtaining a change trend of the data bit width; The data variation range of the data to be quantized is determined according to the variation range of the point position and the variation trend of the data bit width.

18. The method according to claim 17, characterized in that Determining a target iteration interval according to a data variation range of the data to be quantified further includes: The target iteration interval is determined based on the obtained first error and second error, wherein the first error is used to characterize the variation range of the point position, and the second error is used to characterize the variation trend of the data bit width.

19. The method according to claim 18, characterized in that The determining the target iteration interval according to the obtained first error and the second error includes: Taking the maximum value between the first error and the second error as the target error; The target iteration interval is determined according to the target error, wherein the target error is negatively correlated with the target iteration interval.

20. The method according to claim 18 or 19, characterized in that The second error is determined according to the quantization error; The quantization error is determined according to the data to be quantized in the current test iteration and the quantized data of the current test iteration, and the second error is positively correlated with the quantization error.

21. The method according to claim 12, wherein The method is used for training or fine-tuning a neural network, and the method further comprises: When the current iteration is greater than the first preset iteration, a target iteration interval is determined according to the data variation amplitude of the data to be quantized, and the quantization parameter is adjusted according to the target iteration interval.

22. The method according to claim 21, characterized in that The method further comprises: When the current iteration is less than or equal to the first preset iteration, the first preset iteration interval is used as the target iteration interval, and the quantization parameter is adjusted according to the first preset iteration interval.

23. The method according to claim 21 or 22, characterized in that The method further comprises: When the current iteration is greater than or equal to a second preset iteration, the second preset iteration interval is used as the target iteration interval, and the quantization parameter is adjusted according to the second preset iteration interval; The second preset iteration is greater than the first preset iteration, and the second preset iteration interval is greater than the first preset iteration interval.

24. The method according to claim 23, wherein The method further comprises: When the convergence degree of the neural network meets a preset condition, it is determined that the current test iteration is greater than or equal to a second preset iteration.

25. The method according to claim 23, wherein The method further comprises: When the current iteration is greater than or equal to a second preset iteration and the second error is greater than a preset error value, an iteration interval is determined according to a data variation amplitude of the data to be quantized, so as to redetermine the data bit width according to the iteration interval.

26. A device for adjusting the data bit width in a convolutional neural network layer during a neural network operation, characterized in that: The device comprises: an acquisition module, configured to acquire a data bit width for performing a quantization process on data to be quantized, wherein the data to be quantized includes at least one of neurons, weights, gradients, and biases, and the data bit width indicates a data bit width of quantized data of the data to be quantized after the quantization process; a quantization module, configured to perform quantization processing on a set of data to be quantized based on the data bit width, so as to convert the set of data to be quantized into a set of quantized data, wherein the set of quantized data has the data bit width; a determination module, configured to compare the set of data to be quantized with the set of quantized data to determine a quantization error associated with the data bit width; an adjusting module, configured to adjust the data bit width based on the determined quantization error; and An application module, configured to apply the adjusted data bit width during quantization in the convolutional neural network layer; The adjustment module includes: a comparing module, configured to compare the quantization error with a preset threshold, wherein the preset threshold comprises at least one of a first threshold and a second threshold; and A bit width adjustment module, configured to adjust the data bit width according to a result of the comparison; The bit width adjustment module includes: an increasing module, configured to increase the data bit width in response to determining that the quantization error is greater than or equal to a first threshold; The neural network is used for one or more of the following: image recognition or speech recognition.

27. The device according to claim 26, characterized in that The comparison module includes: an interval determination module, configured to determine a quantization interval according to the data bit width; and The error determination module is configured to determine the quantization error according to the quantization interval, the set of quantized data, and the set of data to be quantized.

28. The device according to claim 27, characterized in that The error determination module includes: a quantization module, configured to perform inverse quantization on the set of quantized data according to the quantization interval to obtain a set of inverse quantized data, wherein a data format of the set of inverse quantized data is the same as a data format of the set of data to be quantized; and The quantization error determination module is used to determine the quantization error according to the set of inverse quantized data and the set of data to be quantized.

29. The device according to claim 26, characterized in that The adding module includes: The step-length increasing module is configured to increase the data bit width according to a first preset bit width step to determine an adjusted data bit width.

30. The device according to claim 29, characterized in that: The quantization module is further configured to perform a quantization process on the set of data to be quantized based on the adjusted data bit width, so as to convert the set of data to be quantized into another set of quantized data, wherein the another set of quantized data has the adjusted data bit width; as well as The determination module is further configured to compare the set of data to be quantized with the other set of quantized data, and determine another quantization error associated with the adjusted data bit width, until the another quantization error is smaller than the first preset threshold.

31. The device according to claim 30, characterized in that The device is called iteratively.

32. The device according to claim 26, characterized in that The adjustment module also includes: A reducing module is configured to reduce the data bit width in response to determining that the quantization error is less than or equal to the second threshold.

33. The device according to claim 32, characterized in that The reduction module includes: The step size reduction module is configured to reduce the data bit width according to a second preset bit width step size to determine an adjusted data bit width.

34. The device according to claim 33, characterized in that: The quantization module is further configured to perform a quantization process on the set of data to be quantized based on the adjusted data bit width, so as to convert the set of data to be quantized into another set of quantized data, wherein the another set of quantized data has the adjusted data bit width; as well as The determining module is further configured to determine another quantization error associated with the adjusted data bit width based on the set of data to be quantized and the another set of quantized data, until the another quantization error is greater than the second preset threshold.

35. The device according to claim 34, characterized in that The device is called iteratively.

36. The device according to claim 26, characterized in that The adjustment module also includes: A maintaining module is configured to maintain the data bit width in response to determining that the quantization error is between the first threshold and the second threshold.

37. The device according to any one of claims 26 to 36, characterized in that The device further comprises: an updating module, configured to update a quantization parameter used for performing quantization processing on the set of data to be quantized based on the set of data to be quantized and the adjusted data bit width; and The quantization module is further configured to perform quantization processing on the set of data to be quantized based on the updated quantization parameter.

38. The device according to claim 26, characterized in that The device further comprises: Amplitude module, used to obtain the data variation amplitude of the data to be quantified; The interval module is configured to determine a target iteration interval according to a data variation range of the data to be quantized, so as to adjust the data bit width according to the target iteration interval, wherein the target iteration interval includes at least one iteration.

39. The device according to claim 38, characterized in that The amplitude module includes: The point position module is used to obtain the variation range of the point position, wherein the variation range of the point position is used to represent the data variation range of the data to be quantified, and the variation range of the point position is positively correlated with the data variation range of the data to be quantified.

40. The device according to claim 39, characterized in that The point location module includes: A first mean module is configured to determine a first mean based on a point position corresponding to a previous test iteration before a current test iteration and a point position corresponding to a historical iteration before the previous test iteration, wherein the previous test iteration is a test iteration corresponding to a previous iteration interval before the target iteration interval; a second mean value module, configured to determine a second mean value based on a point position corresponding to the current test iteration and a point position of a historical iteration before the current test iteration; wherein the point position corresponding to the current test iteration is determined based on a target data bit width corresponding to the current test iteration and the data to be quantized; A first error module is configured to determine a first error based on the first mean and the second mean, where the first error is used to characterize a variation range of the point position.

41. The device according to claim 40, characterized in that The spacing module comprises: A first interval module is configured to determine the target iteration interval according to the first error, where the target iteration interval is negatively correlated with the first error.

42. The device according to claim 39, characterized in that The amplitude module further includes: A trend module, used to obtain the change trend of the data bit width; The data variation amplitude module is used to determine the data variation amplitude of the data to be quantized according to the variation amplitude of the point position and the variation trend of the data bit width.

43. The device according to claim 42, characterized in that The spacing module further comprises: The iteration interval module is used to determine the target iteration interval based on the obtained first error and second error; wherein the first error is used to characterize the variation range of the point position, and the second error is used to characterize the variation trend of the data bit width.

44. The device according to claim 43, characterized in that The iteration interval module includes: a target error module, configured to take the maximum value of the first error and the second error as a target error; A target iteration interval module is configured to determine the target iteration interval according to the target error, wherein the target error is negatively correlated with the target iteration interval.

45. The device according to claim 43 or 44, characterized in that The second error is determined according to the quantization error; The quantization error is determined according to the data to be quantized in the current test iteration and the quantized data of the current test iteration, and the second error is positively correlated with the quantization error.

46. The device according to claim 37, characterized in that The apparatus is used for training or fine-tuning a neural network, and the apparatus further comprises: The first iteration module is configured to determine a target iteration interval according to a data variation range of the data to be quantized when the current iteration is greater than a first preset iteration, and adjust the quantization parameter according to the target iteration interval.

47. The device according to claim 46, characterized in that The device further comprises: The second iteration module is configured to use the first preset iteration interval as the target iteration interval when the current iteration is less than or equal to the first preset iteration interval, and adjust the quantization parameter according to the first preset iteration interval.

48. The device according to claim 46 or 47, characterized in that The device further comprises: a third iteration module, configured to, when the current iteration is greater than or equal to a second preset iteration, use the second preset iteration interval as the target iteration interval, and adjust the quantization parameter according to the second preset iteration interval; The second preset iteration is greater than the first preset iteration, and the second preset iteration interval is greater than the first preset iteration interval.

49. The device according to claim 48, characterized in that The device further comprises: The convergence module is configured to determine that the current test iteration is greater than or equal to a second preset iteration when the convergence degree of the neural network meets a preset condition.

50. The device according to claim 48, characterized in that The device further comprises: When the current iteration is greater than or equal to a second preset iteration and the second error is greater than a preset error value, an iteration interval is determined according to a data variation amplitude of the data to be quantized, so as to redetermine the data bit width according to the iteration interval.

51. A computer-readable storage medium, characterized in that A computer program is stored thereon, which implements the method according to any one of claims 1 to 25 when the program is executed.

52. An artificial intelligence chip, characterized in that: The chip comprises the device for processing data according to any one of claims 26 to 50.

53. An electronic device, characterized in that: The electronic device includes the artificial intelligence chip according to claim 52.

54. A board, characterized in that: The board includes: a storage device, an interface device, a control device, and an artificial intelligence chip according to claim 52; Wherein, the artificial intelligence chip is connected to the storage device, the control device and the interface device; The storage device is used to store data; The interface device is used to realize data transmission between the artificial intelligence chip and external equipment; and The control device is used to monitor the status of the artificial intelligence chip.

55. The board according to claim 54, wherein: The memory device includes: a plurality of groups of memory units, each group of memory units is connected to the artificial intelligence chip via a bus, and the memory units are DDR SDRAM; The chip includes: a DDR controller for controlling data transmission and data storage of each of the storage units; The interface device is a standard PCIE interface.

Images