Quantization for performing machine learning model operations

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing operations represented by a neural network including a plurality of neural network layers. The method includes receiving a calculation result of a multiplication operation between a node input of a network layer of the neural network and a corresponding node weight of the network layer. The calculation result has first precision. The calculation results are processed with a particular activation function to generate a layer output. The layer output has a second accuracy that is lower than the first accuracy. The particular activation function is determined based on a quantization process that employs a value of power that is not a multiple of 2. When the calculation result is not zero, the specific activation function generates a non-zero output. A neural network output corresponding to the network input is generated based on the layer output.

Description

Technical Field

[0001] This application relates to quantization operations for efficiently performing operations represented by machine learning models, and more particularly to applying quantization techniques to efficiently perform node computations in neural networks. Background Technology

[0002] A neural network is a machine learning model that employs one or more non-linear units to predict outputs from received inputs. Some neural networks, in addition to an output layer, also contain one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0003] This application describes a technique for efficiently performing node operations in one or more layers of a neural network using one or more quantization techniques. Specifically, the technique generally involves applying a specific activation function to one or more nodes in one or more layers of a neural network. The specific activation function first includes a non-power-of-two (NPOT) quantization scheme applied to the node input. This technique can reduce the computational cost, memory footprint, and power consumption of performing trained neural network operations during the inference phase, while maximizing the utilization of the representable range of a specific data structure and meeting specific accuracy requirements. Furthermore, the specific activation function also includes a specific parameterized ReLU (Rectified Linear Unit) function incorporating NPOT quantization, so that both the NPOT quantization scheme and the parameterized ReLU function can constitute a combined quantization scheme for node operations. Such quantization schemes can further improve the efficiency of performing neural network operations without significantly increasing the number and type of algebraic operations, memory footprint, and power consumption required to perform network layer node operations. Furthermore, the technique may also include one or more global parameters or coefficients for modifying the specific activation function, so that the technique can enable or disable the NPOT quantization function and / or parameterized ReLU function according to different usage requirements of the neural network.

[0004] One aspect of the subject matter described in this application can be implemented by a method comprising operations for performing an operation represented by a neural network comprising one or more neural network layers. The method includes: receiving a computational result of a multiplication operation between node inputs of the network layers of the neural network and the weights of corresponding nodes of the network layers. The computational result has a first precision.

[0005] The method further includes processing the computation result with a specific activation function to generate a layer output. The layer output has a second precision lower than the first precision. The specific activation function is determined based on the NPOT quantization scheme and a parameterized ReLU function. To generate the layer output, the method further includes performing at least one of the following operations: a pre-shift operation, a post-shift operation, a rounding operation, a clip operation, a cut-off operation, an add-shift operation, or other suitable operations.

[0006] Other embodiments of this aspect include corresponding computer systems, apparatuses, computer program products, and computer programs recorded on one or more computer storage media, each configured to perform the operational steps of the method. A system comprising one or more computers may be configured to perform a specific operation or action by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform the operation during operation. One or more computer programs may be configured to perform a specific operation or action by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation.

[0007] Specific embodiments of the subject matter described in this application can be implemented to achieve one or more of the following advantages. The technology can improve the efficiency and robustness of performing neural network operations while meeting the specific accuracy required for neural network usage.

[0008] Typically, quantization schemes are applied to node operations (e.g., by employing specific activation functions) to reduce the number of significant bits representing the values ​​generated by node operations on one or more nodes in one or more layers of a neural network. Quantization schemes generally reduce the computational cost and memory footprint of performing neural network operations, especially for deep neural networks with millions or even billions of model parameters, at the cost of reduced numerical precision due to the use of fewer significant bits. However, this reduction in precision is negligible if the quantized value is sufficiently accurate according to the required specifications.

[0009] Compared to traditional quantization schemes, this technique can further improve the efficiency of performing neural network operations. More specifically, traditional quantization schemes or operations typically have scaling factors that are multiples of 2 (e.g., 2, 4, 8, etc.). Scaling factors are generally used to narrow the representable value range of a particular data structure or data type. For example, a group of integer values ​​ranging from -100 to 100 can be represented using a signed 8-bit integer data type (i.e., INT8), because INT8 can represent values ​​between -128 and 127. However, for the numerical range of -100 to 100 in this example, the representable value range of -128 to -101 and the representable value range of 101 to 127 in INT8 will never be used, which is typically a waste of the representable value range. Since the ratio of 128 to 100 is 1.28, which is greater than 1 but less than 2, it is not possible to simply use power-of-2 quantization to map the integer value to a data type with one less bit (e.g., a signed 7-bit integer data type, INT7). INT7 can only represent values ​​between -64 and 63, and cannot cover the entire possible range of values ​​from -100 to 100.

[0010] To address the aforementioned problem, the technique implements a quantization scheme based on a non-power-of-two (NPOT) scaling factor (e.g., a scaling factor ranging from 1 to 2 (i.e., [1,2])) to map values ​​within the current value range to a value range representable by a predetermined data structure or data type. Furthermore, before performing the NPOT quantization operation, the technique may perform bias operations to improve data symmetry, and perform shift operations before or after the NPOT quantization operation to reduce the data volume. This further improves computational efficiency and reduces the computational cost of performing neural network operations.

[0011] It is important to note that integer data types or formats can be used to represent non-integer values, depending on the data representing the decimal point position. The data representing the decimal point position can be calculated during the calculation and stored in the memory. Other bit depth and format types or data types can be used depending on different calculation requirements. For example, the formatting size can include 8 bits, 10 bits, 12 bits, 16 bits, 24 bits, or other suitable bit depths, and the data type or format can include various floating-point and integer formats depending on different formatting requirements.

[0012] As described above, systems implementing the aforementioned techniques can further improve the efficiency of performing neural network operations by combining parameterized ReLU functions with specific node quantization schemes (e.g., node activation functions using NPOT quantization). Parameterized ReLU functions offer advantages over conventional ReLU functions because they include an adjustable slope (hereinafter also referred to as the "second coefficient") for the third quadrant. This adjustable slope is typically non-zero and can therefore be learned when training the neural network using training samples. Furthermore, parameterized ReLU functions can improve the convergence of the training process because even smaller gradients in the third quadrant contribute to stabilizing the backpropagation process during training. Further, parameterized ReLU functions can improve output accuracy by mitigating dead neurons (e.g., neurons or nodes that generate only zero outputs for all inputs). Moreover, combining parameterized ReLU functions with quantization schemes for one or more nodes in one or more layers does not significantly increase the computational cost required to perform neural network operations, nor does it change the type of computation required to perform the operations represented by the neural network and / or require new types of computation. Conversely, parameterized ReLU functions essentially involve one or more multiplication operations, which are largely the same types of operations required for quantization, thus reducing computational cost and memory footprint. Since the techniques used to implement quantization already introduce additional multiplication operations for quantization, combining parameterized ReLU functions with node NPOT quantization does not significantly increase computational cost.

[0013] Furthermore, the technique is robust to various neural networks with diverse structures. More specifically, the technique includes one or more adjustable global parameters or coefficients for modifying node activation functions, enabling or disabling the NPOT quantization scheme and / or parameterized ReLU function based on different requirements for performing neural network operations. For example, by disabling one or more global parameters or coefficients, the NPOT quantization scheme and parameterized ReLU function can be configured to represent only the leaking ReLU function. As another example, by disabling a first coefficient associated with the adjustable slope, the quantization scheme and parameterized ReLU function can be configured to represent only a single NPOT quantization scheme.

[0014] Furthermore, the technique is robust to different input data from different channels. More specifically, the technique may include various shift operations to ensure the accuracy and consistency of data generated through different channels in different layers.

[0015] Details of one or more embodiments of the subject matter described in this application are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0016] Figure 1 An example quantization system configured to process input data using a specific activation function is shown.

[0017] Figure 2A An example of a numerical distribution is shown.

[0018] Figure 2B Another example of a numerical distribution is shown.

[0019] Figure 3A An example activation function with a specific quantization scheme and parameterized ReLU function is shown.

[0020] Figure 3B It shows from Figure 3A Example of a non-power-of-two (NPOT) quantization scheme derived from the example activation function derivation.

[0021] Figure 3C It shows from Figure 3A The example activation function is used to derive the standard ReLU function.

[0022] Figure 4 It shows Figure 1 Example of an operation sequence performed by a sample quantization system.

[0023] Figure 5 An example of an interlayer structure of a neural network containing one or more additive channels is shown.

[0024] Figure 6 This is a flowchart of an example process for processing input data using a specific activation function.

[0025] The same reference numerals and symbols in the accompanying drawings represent the same elements. Detailed Implementation

[0026] The technique relates to quantization operations for performing node operations in one or more layers of a neural network. Typically, node operations in a network layer generally include addition and multiplication operations. This technique improves the efficiency of performing network layer node operations and reduces computational and memory costs by implementing non-power-of-two (NPOT) quantization. More specifically, the NPOT quantization scheme involves mapping one or more computation results stored in a higher-precision first format (e.g., using more significant digits or a larger or more powerful data structure) to a lower-precision second format (e.g., using fewer significant digits or a smaller or less powerful data structure) based on a scaling value greater than or equal to 1 and less than 2. Although the NPOT quantization scheme may increase the number of multiplication operations required to perform network layer node operations because it may introduce additional multiplication operations into the original neural network operation, it reduces overall computational cost and memory footprint by using a lower-precision data type or format to represent the numerical values.

[0027] To further leverage the advantages of NPOT quantization, the technique implements specific node operations beyond the standard ReLU function for one or more nodes in one or more layers. For example, specific node operations may include leaked ReLU functions or parameterized ReLU functions. These specific node operations can further improve the accuracy and convergence of performing network layer node operations because leaked ReLU functions and parameterized ReLU functions have non-zero first derivatives, which help improve the convergence of the backpropagation phase of the training process.

[0028] Nevertheless, certain node operations, such as leaky or parameterized ReLU functions, may introduce additional multiplication operations to the execution of node operations in a neural network. However, the overall impact of these additional operations on computation and memory usage is minimal or negligible, partly because these additional operations are all multiplication operations, the same type as those introduced by the NPOT quantization scheme. It is worth noting that, in terms of computational cost and memory usage, multiplication operations are generally preferred over addition (or subtraction) operations in the operations represented by neural networks.

[0029] Figure 1An example quantization system 100 configured to process input data 110 using a specific activation function is illustrated. Typically, the quantization system 100 may be implemented on one or more computers or processors at one or more locations. The one or more computers or processors may be wirelessly or wiredly coupled to each other. The one or more computers or processors may include one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or other suitable types of processors. For simplicity, the quantization system 100 is also referred to as System 100 in the following description.

[0030] like Figure 1 As shown, system 100 may include one or more engines for processing input data 110 and generating output data 180. The input data 110 typically includes one or more computational results of node inputs and node weights from one or more nodes in one or more layers of a neural network. For example, the computational results may include the multiplication result of the node input and node weights of a node in a certain layer. The node input of the corresponding node in the current layer may be the node output (e.g., a scalar, vector, or matrix of node outputs) of a node in the layer immediately preceding the current layer, and the node weights may be weights associated with the corresponding node (e.g., a scalar, vector, or matrix of weights). In some embodiments, the computational result may be the sum of one or more multiplication results. For example, multiplying the node input vector of a node in a certain layer by the node weight matrix results in a vector form, and the computational result is the sum of the vectors of the multiplication results.

[0031] Input data 110 can be stored in a data type or format with first precision. For example, input data 110 can be stored as an integer type, such as (signed or unsigned) INT12 integer, INT14 integer, INT16 integer, INT24 integer, or other suitable integer types. It should be noted that, depending on different calculation requirements, input data 110 can also be stored as a floating-point type or format with a specific number of bits.

[0032] The computation result of input data 110 is further provided as input to one or more node activation functions of the current node in the current layer. The node activation functions typically perform a nonlinear transformation on the computation result before providing it as the output of the current node to the corresponding node in the next adjacent layer of the neural network.

[0033] In some implementations, input data 110 may include node inputs and corresponding node weights for the current layer. The quantization system 100 may include a multiplication unit configured to process the node inputs and node weights by multiplying (and optionally summing) the node inputs and node weights to generate the computation result. For specific hardware, node output data and node weights may be stored and received by the multiplication unit and / or the quantization system 100 in a first number of bits (e.g., bit width) with a first precision (e.g., 8-bit INT8), and the computation result may be stored in a second number of bits with a second precision (e.g., 16-bit INT16 or 24-bit INT24). For simplicity, input data 110 as used below refers by default to the computation result based on the node weights and node inputs for the current layer.

[0034] Output 180 typically includes the node outputs of the node activation functions corresponding to one or more nodes in the current layer. The outputs of these node activation functions are also referred to as the node outputs corresponding to one or more nodes in the current layer. Output 180 is then provided as input to one or more nodes in subsequent layers of the neural network.

[0035] For a given hardware or computing unit, output data 180 is stored in a data type with the same precision level as input data 110. For example, in the case where quantization system 100 has a multiplication unit, the input data includes node weights and corresponding node inputs of 8-bit data (e.g., INT8 or unsigned 8-bit integers (UINT8)), and the output data accordingly has the same 8-bit number of bits (e.g., INT8 or UINT8). However, in some cases, when the input data includes computation results generated by the node inputs and node weights of the current layer, the input data may be stored in a data type or format with higher precision and a larger number of bits (e.g., INT16 or INT24), and the output data 180 may be stored in a data type or format with lower precision (e.g., INT8 or UINT8). It should be noted that, depending on different computational requirements or hardware design, output data 180 may be stored as an integer type, a floating-point type, or other suitable type with a specific number of bits.

[0036] System 100 may include a bias engine 120 configured to process input data 110 by performing bias operations on it to generate bias data 125. The bias engine 120 first determines whether to perform one or more bias operations on the input data 110 based on one or more criteria. These criteria may include at least one of the following: whether the input data 110 has a symmetric numerical distribution; whether the mean or median of the input data 110 deviates from a predetermined value (e.g., zero, a non-zero positive value, etc.). In response to determining that the input data 110 has an asymmetric numerical distribution and / or that the mean or median of the input data 110 deviates from a predetermined value, the bias engine 120 performs one or more bias operations on the input data 110 based on the bias value. The bias value is determined based on the distribution of all possible values ​​of the input data 110. For example, for a computation result of the input data 110, the bias value is determined based on the probability distribution of all values ​​of the computation result calculated from the corresponding node inputs and node weights.

[0037] As a more specific example Figure 2A An example of a numerical distribution 210 is shown, which represents each possible value of the input data 110 (or the corresponding calculation result) with a corresponding probability. The numerical distribution 210 is approximately symmetrical on an axis with a probability of zero. Figure 2A The horizontal axis represents the numerical value of the input data (or calculation result), and the vertical axis represents the probability of a specific value. Therefore, according to... Figure 2A Values ​​closest to zero have the highest probability; the further a value is from zero, the lower its probability. The term "roughly symmetric" in this specification is generally distinguished from "symmetric," where roughly symmetric does not require strict symmetry. Figure 2A As shown, although the numerical distribution 210 is not strictly symmetrical around the axis of zero probability, the mean or median of the numerical distribution 210 is most likely to be zero, and the probabilities are roughly mirrored on both sides of the axis of zero probability.

[0038] Furthermore, the numerical distribution 210 represents the probability of a value falling within and outside the interval between boundary values ​​#1 (220) and #2 (225). Values ​​falling outside the interval refer to values ​​below range 230 (e.g., less than boundary value #1 (220)) and values ​​above range 235 (e.g., greater than boundary value #2 (225)). Since the numerical distribution 210 is approximately symmetrical and has a mean / median of zero, the possible values ​​of the numerical distribution 210 roughly cover the full resolution 240 of the interval. In other words, it is more ideal for most values ​​of the numerical distribution 210 to be covered within the interval, as the data structure used to represent values ​​within the interval between boundary values ​​#1 (220) and #2 (225) can be fully utilized, improving the efficiency and utilization of the data structure and memory. The bias value and bias data 125 should have the same data type or format as the calculation result based on the node weights and node inputs of the current layer (e.g., input data 110). Accordingly, the bias value and bias data 125 may be represented by a data type or format, including integers (e.g., INT12, INT16, INT24, etc.), floating-point types, or other suitable data types or formats.

[0039] Typically, system 100 may use the INT data type or format to represent floating-point values. More specifically, system 100 may use the INT data type to represent significant bits in floating-point data and store data representing the decimal point position, the exponent part, or both, which may be stored in one or two bits of the INT data type or in the storage unit 190 of system 100.

[0040] Figure 2B Another example of numerical distribution 250 is shown. Similar to numerical distribution 210, numerical distribution 250 represents each possible value of input data 110 (or a corresponding calculation result) with a corresponding probability. However, the probability distribution of numerical distribution 250 is approximately symmetrical about an axis of reference value 285, for example, reference value 285 could be the mean, median, or expected value of all possible values ​​of numerical distribution 250. The reference value 285 is biased by a value 280 greater than zero. The bias value 280 is determined based on the difference between the reference value 285 and a value that is approximately symmetrical to numerical distribution 250. In this case, even though numerical distribution 250 appears approximately symmetrical around the axis of reference value 285, it is not approximately symmetrical around the zero value. Accordingly, numerical distribution 250 only covers a portion of the full range of representable values, which lies within the interval between boundary value #1 (260) and boundary value #2 (265). Figure 2B The full range of representable values ​​is also called full resolution 290, and the portion of the full range of representable values ​​covered by the numerical distribution 250 is also called reduced resolution 295.

[0041] To improve the utilization of the full resolution 290, the bias engine 120 is configured to determine whether the input data 110 (or the calculation results based on the current layer node weights and node inputs) needs biasing based on one or more criteria. The one or more criteria include at least one of whether the numerical distribution of the input data 110 is approximately symmetrical, or whether the mean or median of the numerical distribution deviates from a predetermined value (e.g., zero). In response to determining that the input data 110 meets one or more criteria, the bias engine 120 can shift all values ​​in the input data 110 (and corresponding calculation results) by adding or subtracting a bias value determined based on the numerical distribution. Figure 2B In this configuration, the bias engine 120 is configured to bias the input data 110 by shifting all values ​​in the input data 110 to the left by a bias value 280. This results in a biased numerical distribution 255 that is approximately symmetrical around zero, allowing the system to represent most possible values ​​of the biased numerical distribution 255 using full resolution 290.

[0042] The bias engine 120 is configured to bias the input data 110 based on the bias value 280 to generate bias data 125. When the bias engine 120 determines not to perform a bias operation on the input data 110, it can directly provide the input data 110 as output. Therefore, for simplicity, the bias data 125 usually refers to the output data generated by the bias engine 120, and does not necessarily refer to the data after bias processing by the bias engine 120.

[0043] See again Figure 1 System 100 includes a pre-shift engine 130 for reducing the size of the bias data 125 output by the bias engine 120. Typically, the pre-shift engine 130 is configured to perform a pre-shift operation on the bias data 125 to generate pre-shifted data 135 with a smaller data size than the bias data 125. Accordingly, the precision of the pre-shifted data 135 is lower than the precision of the input data 110 and the bias data 125. The pre-shifted data 135 can be represented by a data type with one, two, five, ten, twelve, or other suitable number of bits less than the input data 110 or the bias data 125. For example, the input data 110 or the bias data 125 can be represented by INT24, and the pre-shifted data 135 can be represented by INT12, INT11, or other suitable data types or formats. In the following description, the pre-shifted data 135 is also referred to as an intermediate result.

[0044] Pre-shift operations performed by the pre-shift engine 130 typically involve reducing data size or precision by removing lower-order bits from the corresponding data type or format. For example, a pre-shift operation can reduce a 24-bit data type to a 12-bit data type by removing the lower 12 bits. Pre-shift operations can also be referred to as power-of-two standard quantization operations. It should be noted that pre-shift operations do not include the NPOT quantization scheme and parameterized ReLU function described above. However, pre-shift operations are generally beneficial because they reduce the data size (or bit width) and memory footprint required for the NPOT quantization scheme and parameterized ReLU function to perform their operations.

[0045] To generate pre-shifted data 135, the pre-shift engine 130 is further configured to perform a pruning operation, modifying values ​​outside the reduction range to corresponding boundary values. For example, for the range [-1000, 999], the pre-shift engine 130 is configured to modify the value 1001 to 999 because 1001 is outside the range and 999 is the corresponding boundary value (e.g., the nearest boundary value). Since 1001 is closest to the upper boundary (i.e., 999), the pre-shift engine 130 is configured to add the value 999 to this set of values. Further details of the operations performed by the pre-shift engine 130 will be provided below. Figure 4 Description. The pre-shifted data 135 is then processed by the quantization engine 150. Further details of the quantization engine 150 will be described below.

[0046] In some implementations where the current network layer is coupled to another network layer (e.g., at least a portion of the output of the current network layer is summed with a portion of the other network layer), system 100 may include an add-shift engine 140 for further aligning the decimal points of the node outputs of the current network layer and the other network layer before merging them. More specifically, the add-shift engine 140 is configured to determine whether the other network layer is an additive layer containing additive channels. An additive layer typically refers to a network layer whose output is configured to be summed with the output of the current network layer before the merged output is processed by the corresponding node activation function. An additive channel contains one or more nodes whose node outputs are summed with the corresponding additive channel in the current network layer.

[0047] In response to determining that another network layer is an addition layer containing an addition channel, the add-shift engine 140 is configured to shift the decimal point of the output of the addition channel of the addition network layer (e.g., appended input data 165) to align with the decimal point of the output of the current network layer (e.g., pre-shifted data 135). For example, to align the decimal point, the add-shift engine 140 may be configured to shift the decimal point position of the appended input data 165 by 1, 2, 3, or other suitable number of bits to generate add-shifted data 145. In some cases where the decimal points of the addition layer output and the current layer output are already aligned, the add-shift engine 140 accordingly does not shift the decimal point of the addition layer. In these cases, the appended input data 165 and the add-shifted data 145 have the same decimal point position.

[0048] Once the decimal points are aligned, the quantization system 100 is configured to add the outputs of the two addition channels of the two layers to generate merged data as input to the quantization engine 150. For example... Figure 1 As shown, the pre-shifted data 135 and the added-shifted data 145 are combined or added to generate merged data 170, which is then provided to the quantization engine 150 for further processing. More details of the added-shift operation performed by the added-shift engine 140 will be provided below. Figure 5 describe.

[0049] The quantization system 100 further includes a quantization engine 150, which is configured to process pre-shifted data 135 or merged data 170 using a specific activation function. The specific activation function includes an NPOT quantization scheme and a parameterized ReLU function. Further details of the specific activation function will be provided below. Figure 3A , 3B And 3C description. The quantization engine 150 is configured to generate quantized data 155 after processing the shifted data 145. In the following description, the quantized data 155 is also referred to as the node activation value.

[0050] By employing the NPOT quantization scheme, system 100 can map numerical values ​​(e.g., pre-shifted data 135 or a value with added shift 145) to the value range of a predetermined data type based on a scaling value that is not a power of 2 (e.g., a scaling value greater than or equal to 1 and less than 2). Accordingly, the NPOT quantization scheme allows quantization engine 150 to maximize the resolution utilization of the predetermined data type, thereby improving the efficiency of performing neural network operations and reducing memory footprint. As an example of the pre-shift operation, the system can remove up to 12 low-order digits (or bits) from the data structure. For an input stored as INT24, the output after the pre-shift operation can be INT12.

[0051] As an example illustrating the NPOT quantization scheme, system 100 can process INT12 inputs generated by a pre-shift operation to produce an INT13 output. System 100 first determines the maximum value in the set of values ​​to be quantized. Assuming the maximum value in this example is 1303 stored in the INT12 data structure, the system can map 1303 to the upper boundary of INT12, 2047. The system determines the NPOT parameters by dividing 2047 by 1303, obtaining the floating-point value 1.57099.

[0052] Furthermore, System 100 employs the rounding technique described herein to round the floating-point value of the NPOT parameter into an integer data structure. For example, System 100 rounds the floating-point value of the NPOT parameter up by multiplying 1.57099 by 128, obtaining the integer 201. The system stores the value 201 to approximate the actual floating-point value of the NPOT parameter. It should be noted that System 100 still uses the INT13 data structure to store the output in order to capture intermediate results exceeding the INT12 boundary values. For example, when the maximum value of the dataset determined by the system (or the user) is not the true maximum value, System 100 can capture those overflow values ​​by using a data structure with one more bit.

[0053] System 100 can use an 8-bit data structure to store the NPOT parameter value, where the first bit represents the integer 1 and the last seven bits represent the fractional part. However, since the NPOT parameter here is always greater than 1, the system can subtract 1 from the actual NPOT parameter (e.g., 1.57099) to reduce the memory bandwidth required to store the NPOT parameter. In this way, the system only needs 7 bits instead of 8 bits to store the NPOT parameter. In this example, system 100 stores 73 instead of 201 in a register. When an operation requires the NPOT parameter value, system 100 adds 128 to 73 to obtain 201, and uses 201 / 128 to approximate the floating-point value of the NPOT parameter. It should be noted that, in the following description, the aforementioned NPOT parameter is also referred to as the first coefficient associated with the NPOT scheme.

[0054] For an input with a maximum value of 1303, the system can generate an output value of 2046 by multiplying 1303 by 201 / 128 and rounding up. If the input value is different from the maximum value (e.g., 100), the system can obtain the output value (e.g., 157) by multiplying the input (e.g., 100) by 201 / 128.

[0055] A corresponding example of the back-shift operation is described below. Furthermore, more details of the NPOT quantization scheme will be discussed below. Figure 3A , 3B And 3C description.

[0056] System 100 includes a back-shift engine 160 configured to process quantized data 155 to generate output data 180. The operations performed by the back-shift engine 160 are similar to those performed by the pre-shift engine 130. More specifically, the back-shift engine 160 is configured to reduce the size or precision of the quantized data 155 by removing the lower bits of the data type used to store the quantized data 155. For example, if the quantized data 155 is stored as an INT13 data type, the back-shift engine 160 can remove the lower 5 bits of the INT13 data type to generate an output stored as INT8. The operations performed by the back-shift engine 160 can also be referred to as standard quantization operations using powers of 2 or multiples of 2. Similar to the above, the back-shift operation does not include operations related to the NPOT quantization scheme and parameterized ReLU function.

[0057] To generate output data 180, the post-shift engine 160 is further configured to perform a rounding operation, rounding up values ​​greater than 0.5 in the least significant bit. Furthermore, similar to the pre-shift engine 130, the post-shift engine 160 is also configured to perform a clipping operation, modifying values ​​outside the reduction range to the corresponding boundary values ​​of the reduction range. For example, if the rounded value exceeds the representable range [-128, 127] (e.g., 128), the post-shift engine 160 is configured to modify 128 to 127 (i.e., the boundary value closest to 128). The post-shift engine 160 is configured to provide this set of values ​​as output data 180. As another example, for the value -130, which exceeds the range [-128, 127], the post-shift engine 160 is configured to modify -130 to -128 (i.e., the boundary value closest to -130). Further details of the operations performed by the post-shift engine 160 will be provided below. Figure 4 describe.

[0058] Furthermore, system 100 may be communicatively coupled to storage unit 190. Storage unit 190 may be a local storage unit or a remote storage unit of quantization system 100. In some cases, storage unit 190 is typically configured to store parameters of quantization system 100. For example, storage unit 190 may store model parameters (e.g., node weights) of a neural network. Storage unit 190 may also provide these stored parameters to system 100 to perform neural network operations. In addition, storage unit 190 may further store parameters of the NPOT quantization scheme and parameterized ReLU function, and provide these parameters to system 100 to perform pre-shift and post-shift operations. Storage unit 190 is further configured to store data representing the decimal point position and / or exponent data used for quantization. In some embodiments, storage unit 190 may optionally be configured to store input data 110 and provide it to system 100, or temporarily store output data 180 (e.g., as a buffer), or both.

[0059] System 100 may be communicatively coupled to server 195. Server 195 typically receives user requests to process input data 110 using quantization system 100. In some cases, server 195 may receive and further process output data 180 to use the output generated by a neural network. In some cases, server 195 may generate instructions that, once executed by quantization system 100, enable or disable the NPOT quantization scheme and parameterized ReLU function via global parameters, and cause system 100 to process input data 110 with or without enabling or disabling the NPOT quantization scheme, parameterized ReLU function, or both.

[0060] Figure 3A An example activation function 300 is shown, employing a specific quantization scheme and parameterized ReLU function. For example... Figure 3A As shown, the activation function 300 is expressed as follows.

[0061] Formula (1),

[0062] in, This represents the value to be processed by the node activation function, and the output is... This indicates the node output corresponding to the given node. The first coefficient associated with the NPOT quantization scheme, and ; The second coefficient is associated with the parameterized ReLU function, and In fact, the second coefficient Based on the first coefficient The value is determined, that is ,in The coefficients are those of the standard parameterized ReLU function as described below.

[0063] ,in Formula (2).

[0064] It should be noted that, The value can be 0 or 1. Furthermore, The value of can be zero but cannot be two. For example, as mentioned above, if It is 0.1 and If the value is 1.57099, then the coefficient is... The value is 0.157099, which can be stored by System 100 in an 8-bit data structure. For example, System 100 can... The floating-point value is multiplied by 128, and the result, 20, is stored in an integer data structure. System 100 can approximate this by dividing 20 by 128. Therefore, system 100 can generate the output value corresponding to the negative input value by multiplying the negative input value by 20 / 128. For example, for the input value -100, the output value can be obtained by multiplying -100 by (20 / 128) to get -16. If The value is 1 (i.e. If the input is -100 multiplied by (201 / 128), the output is -157.

[0065] Furthermore, formula (1) further defines the threshold. , making when When, set In other words, the output value Limited to a threshold It should be noted that the threshold mentioned is different from the truncation operation described above.

[0066] As mentioned above, activation function 300 integrates the NPOT quantization scheme and the parameterized ReLU function. The additional operation introduced into the original neural network operation is the first coefficient. With numerical values The multiplication operation, and the second coefficient With numerical values Multiplication operations. Although the additional multiplication operations increase the total number of operations, multiplication requires fewer computational resources than addition, making it more preferred; and due to the implementation of pre-shift operations and the NPOT quantization scheme, the overall computational cost and memory footprint are reduced. Furthermore, because the parameterized ReLU function applies to all... All of them have non-zero first-order derivatives, and the overall convergence of neural network training is also improved.

[0067] like Figure 3A As shown, the activation function 300 is divided into three parts: Part I (310), Part II (320), and Part III (330). Part I (310) usually refers to... The interval greater than zero. In part I (310), the slope of activation function 300 is determined by the first coefficient. Definition, the Greater than (or equal to) 1 and less than 2. Part I (310) is usually associated with NPOT quantization because the numerical value With node output The mapping between them is based on the first coefficient. Implementation. Part II (320) usually refers to The third quadrant interval less than or equal to zero. As mentioned above, the second coefficient... Non-zero and related to the parameterized ReLU function. Part III (330) typically refers to all values ​​greater than the threshold. of exist The truncated interval.

[0068] By adjusting the first coefficient and / or the second coefficient System 100 (or pre-shift engine 130 and / or post-shift engine 160) can enable or disable NPOT quantization and parameterized ReLU functions. In some implementations, system 100 may further include one or more global parameters for enabling or disabling the NPOT quantization scheme and parameterized ReLU functions.

[0069] Figure 3B It shows from Figure 3A The example activation function derivation yields a non-power-of-two (NPOT) quantization scheme 340. For example... Figure 3B As shown, by using the first coefficient Set to equal the second coefficient The parameterized ReLU function can be disabled, causing system 100 (or quantization engine 150) to execute only the NPOT quantization scheme 340. System 100 can further implement a threshold. For values ​​greater than the threshold of The output value is subject to an upper limit.

[0070] Furthermore, Figure 3C It shows from Figure 3A The standard ReLU function 380 is derived from the example activation function derivation. Figure 3C As shown, by using the first coefficient Set to 1, formula (1) is expressed as follows.

[0071] Formula (3)

[0072] The function derived from formula (3) In fact, it is the standard parametric ReLU function 380, where That is, formula (2). System 100 can further implement the threshold. For values ​​greater than the threshold of The output value is subject to an upper limit.

[0073] Accordingly, the NPOT quantization scheme and parameterized ReLU function can be easily enabled or disabled by adjusting the first coefficient and / or the second coefficient. Therefore, the NPOT quantization scheme and parameterized ReLU function executed by system 100 and its corresponding engine are dynamic and robust to different requirements for performing neural network operations.

[0074] Figure 4 It shows Figure 1Example 400 shows the sequence of operations performed by the example quantization system 100.

[0075] System 100 receives calculation result 430 as input data. Calculation result 430 includes one or more multiplication results between the output 410 of the previous layer and the corresponding weight 420 of the current layer. The corresponding weight may represent the node weight of one or more nodes in the current layer. The corresponding weight may be stored in different data structures, such as scalars, vectors, matrices, or other suitable data structures. Calculation result 430 is typically obtained by multiplying the output 410 of the previous layer by the corresponding weight 420 of the current layer, and in some cases, it also includes summing the multiplication results. In these cases, calculation result 430 is typically stored in a data type whose size or precision is greater than that of the output 410 of the previous layer and the weight 420 of the current layer. For example, if the output 410 of the current layer and the weight 420 of the current layer are stored as INT8 or UINT8 data types, the calculation result 430 may be stored as INT16 or INT24 data types.

[0076] In some implementations, as described above, system 100 directly receives the output 410 of the previous layer and the weights 420 of the current layer as input data. In these cases, the output 410 of the previous layer and the weights 420 of the current layer can be received using the same data type as the output data 490, such as INT8 or UINT8.

[0077] System 100 (or bias engine 120) is configured to perform one or more bias operations 440 on the computation result 430. As described above, system 100 is configured to determine whether to bias the computation result 430 based on one or more criteria. These criteria include at least one of the following: whether the numerical distribution of the computation result 430 is symmetric about a certain axis of symmetry; or whether the mean or median of the numerical distribution of the computation result 430 is equal to zero. To bias the computation result 430, system 100 adds or subtracts a bias value to the computation result 430, the bias value being determined based on the numerical distribution of the computation result 430 and stored with the same data size or precision, such as INT16 or INT24. As described above, bias operation 440 can improve the utilization of the resolution of data types used to represent data within a specific numerical range. Further details regarding the numerical distribution and bias operations are described above.

[0078] System 100 (or pre-shift engine 130) is configured to perform one or more pre-shift operations 450. Pre-shift operations 450 typically involve reducing the data size or precision of the bias data output by bias operation 440. For example, pre-shift operation 450 involves removing the least significant bits of the data type storing the bias data, such that the pre-shifted data has one or more fewer bits than the bias data. The output data generated by the pre-shift operation may have a size of 12 bits and be stored as, for example, INT12.

[0079] Typically, as described above, pre-shift operation 450 does not include the execution of the NPOT quantization scheme and parameterized ReLU function. However, pre-shift operation 450 may be referred to as using a standard quantization scheme that employs powers of 2. System 100 may use a 4-bit register to store the instructions for performing pre-shift operation 460, which can be used to determine the lowest number of bits (up to 12 bits) to be removed from a 24-bit data structure.

[0080] The pre-shift operation 450 may include a pruning operation 455 for modifying values ​​that exceed the reduced resolution range of the NPOT quantization scheme. The pruning operation typically includes determining whether a value exceeding the reduced resolution range is an outlier. In response to determining that the value is an outlier, system 100 may modify the outlier to one of the two boundary values ​​of the reduced resolution that corresponds to the outlier. For example, if the outlier is greater than the larger of the two boundary values, system 100 modifies the outlier to the larger of the two boundary values. As another example, if the outlier is less than the smaller of the two boundary values, system 100 modifies the outlier to the smaller of the two boundary values. System 100 provides this set of data as the output of the pre-shift engine 130.

[0081] After performing the pre-shift operation 450 and the pruning operation 455, the system 100 is configured to perform an operation 465 of a specific activation function to process intermediate results and generate quantized data or node activation values. As described above, the specific activation function includes operations of the NPOT quantization scheme and node activation operations based on a parameterized ReLU function. The NPOT quantization scheme and the parameterized ReLU function can be expressed as Equation (1). By setting different values ​​for the first and second coefficients, and / or setting different values ​​for one or more global parameters, the system 100 can enable or disable the NPOT quantization scheme and / or the parameterized ReLU function, and even convert the parameterized ReLU function to a standard parameterized ReLU function or a leaky ReLU function.

[0082] NPOT quantization typically refers to a quantization scheme that uses a scaling factor other than a power of 2, in contrast to standard quantization operations that use a scaling factor of a power of 2. By implementing NPOT quantization, system 100 can more efficiently represent numerical values ​​using data types (e.g., Figure 1The input data is 110 or the calculated result is 430, because NPOT quantization can further improve the utilization of full resolution. In addition, parameterized ReLU function improves convergence by using non-zero slope values ​​for negative inputs, thereby further improving efficiency.

[0083] Instructions for executing the NPOT quantization scheme can be stored in a 7-bit register. More specifically, due to the first coefficient in NPOT quantization... The value is greater than 1, therefore system 100 can be obtained from the first coefficient. Subtract 1 from the middle, and store the fractional part of the first coefficient using 7 bits. System 100 is configured to use 8 bits to store the second coefficient. The decimal part. It should be noted that... and The values ​​can all be stored in 7 bits to represent the corresponding number of decimal places.

[0084] In some implementations, depending on the topology of the neural network, system 100 needs to merge the node multiplication results of the current network layer with the output (or intermediate result) of another network layer before executing the operation of a specific node activation function. In these cases, system 100 first performs an add-shift operation 460 on the output or intermediate calculation result 457 of the other layer to align its decimal point with the decimal point of the pre-shifted data of the current network layer. The instruction for the add-shift operation can be stored in a 2-bit register, which can specify whether to shift the decimal point by 1, 2, 3 bits, etc., or not to shift the decimal point at all. Further details of the add-shift operation 460 will be provided below. Figure 5 describe.

[0085] After aligning the decimal points of the calculation results from different network layers, as described above, system 100 performs an addition operation 463 to sum the pre-shifted data with the added-shifted data to generate merged data. The merged data generated by addition operation 463 is then processed by system 100 using operation 465 with a specific activation function, as described above.

[0086] In some implementations, depending on the different topologies of the neural network, system 100 can add the calculation result 457 of another layer to the output data processed by a specific activation function.

[0087] System 100 is further configured to perform a backward shift operation 470. Similar to the pre-shift operation 450, the backward shift operation 470 is performed by system 100 to further reduce the required data size. For example, the backward shift operation can further reduce the data size from a 12-bit data structure to an 8-bit data structure (e.g., INT8 or UINT8). The instructions for the backward shift operation 470 can be stored in a 3-bit register.

[0088] The shift operation 470 further includes a rounding operation 475 and a truncation operation 480. System 100 is configured to perform rounding operation 475, rounding up values ​​greater than 0.5 in the least significant bit. If a value, after being rounded up, exceeds the resolution range determined by the shift operation, system 100 can truncate the rounded result to one of the two boundary values ​​of the data type. For example, for the resolution range [-128, 127] of signed INT8 integers, if a value is quantized to 127.6 and rounded up to 128, the system can modify the rounded value 128 to 127, thereby truncation the rounded data back to the right boundary of INT8.

[0089] The cropping operation 480 is similar to the cropping operation 455 performed in the pre-shift operation 450, except that, in addition to identifying and modifying outliers in the original data set, the cropping operation 480 also includes modifying values ​​that exceed the resolution range after rounding up.

[0090] The output of the shift operation 470 is also referred to as the candidate layer output of one or more nodes in the current layer. System 100 is configured to perform a truncation operation 485 on the candidate layer output. The truncation operation 485 is typically related to the threshold defined in formula (1). Related. As mentioned above, system 100 can determine whether the candidate layer output is greater than the threshold. And in response to determining that the candidate layer output is greater than a threshold Set the candidate layer output as the threshold Output the generated layer.

[0091] Figure 5 An example of the interlayer structure of a neural network 500 containing one or more addition channels 510 and 540 is shown. Figure 5 The operations shown can be performed by a system that implements the technology. For example, Figure 1 After proper programming, System 100 can execute Figure 5 The operations within.

[0092] like Figure 5As shown, neural network 500 includes an addition layer and a convolutional layer. The addition layer includes a copy channel 530 and an addition channel 540. The convolutional layer includes an addition channel 510 and an insertion channel 520. As described above, system 100 merges the outputs of addition channels 540 and 510, and then performs an operation 555 defined by a specific activation function on the merged data. More specifically, system 100 performs a shift-by-1 operation 543, shifting the output of addition channel 510 by 1 bit to reduce the data size from 12 bits (e.g., INT12) to 11 bits (e.g., INT11). This is because the addition operation 550 from the data from the two addition channels 510 and 540 increases the data size by 1 bit. As described above, system 100 performs an addition shift operation 545 to determine whether and how to shift the decimal point of the output data of addition channel 540. For example, system 100 may shift the decimal point by 3 bits to change the data size from 8 bits (e.g., INT8) to 11 bits (e.g., INT11). Subsequently, system 100 performs an addition operation 550, merging the data from addition channels 510 and 540. The output of system 100 after performing addition operation 550 is 12 bits (e.g., INT12). Next, system 100 performs a specific activation function operation 555, a shift operation 560, and a truncation operation 565 in a similar manner to that described above, to generate the output of the output layer addition channel 585. The output of addition channel 585 can be stored in 8 bits (e.g., INT8).

[0093] System 100 also performs a 1-bit shift operation 547, shifting the output of insertion channel 520 by 1 bit to reduce the data size from 12 bits (e.g., INT12) to 11 bits (e.g., INT11). Next, system 100 performs a specific activation function operation 557, a post-shift operation 563, and a truncation operation 567 in a similar manner to those described above to generate the output of output layer insertion channel 590. The output of insertion channel 590 can be stored in 8 bits (e.g., INT8).

[0094] System 100 performs a power-of-2 quantization operation 570 to process the output data of replication channel 530, and then performs a standard ReLU operation 573 to generate node outputs. System 100 then provides the node outputs to the replication channel 580 of the output layer.

[0095] In some implementations, system 100 may determine whether the data type of the node output of one or more channels (e.g., copy channel 530, addition channels 540 and 510, insertion channel 520) differs from the native data type supported by the corresponding computation unit allocated to the computation. In response to determining that the data type of the node output differs from the native data type, system 100 is configured to convert the data type of the node output to the native data type before providing the data as output to the corresponding channel of the output layer. For example, system 100 may determine that the native data type of the output layer node output supported by the corresponding computation unit is an unsigned integer format (e.g., UINT8), while the intermediate computation results of performing the current convolutional layer (and optionally addition layer) operation are stored as signed 8-bit integers (e.g., INT8). In this case, system 100 may convert the data type of the node output to an unsigned integer format (e.g., UINT8) before providing the node output as the output of the corresponding channel of the output layer.

[0096] Figure 6 This is a flowchart of an example process 600 that processes input data using a specific activation function. For ease of illustration, the example process 600 is described as being executed by a system consisting of one or more computers located in one or more locations. For example, Figure 1 The quantization system 100, after being properly programmed, can execute the process 600.

[0097] Typically, the system can perform operations represented by a neural network. The neural network comprises multiple neural network layers. Each layer contains one or more nodes, and each node is associated with one or more node weights and a corresponding node activation function.

[0098] The system receives the calculation result (610) of the multiplication operation between the node input of a certain network layer of the neural network and the corresponding node weight of that network layer. The calculation result can be a scalar or a vector, representing a part of the current network layer node operation. The calculation result has a first precision, which is higher than the precision of storing the node input and node weight. For example, the node input and node weight can be stored as an 8-bit data type (e.g., INT8 or UINT8), and the calculation result can be stored as a 16-bit or 24-bit data type (e.g., INT16 or INT24). It should be noted that, depending on the different requirements for performing neural network operations and / or different hardware designs, the calculation result can be stored in other suitable data types or formats.

[0099] The system processes the computation results using a specific activation function to generate layer output (620). The layer output can be stored as a second-precision data type, which is lower than the first precision. For example, the second precision can have the same data size as the node inputs and node weights (e.g., 8 bits or INT8). It should be noted that, depending on the different requirements for performing neural network operations and / or different hardware designs, the layer output can be stored as other suitable data types or formats.

[0100] The specific activation function is determined based on a quantization process using powers other than 2, and the specific activation function generates a non-zero output when the calculation result is non-zero by using a parameterized ReLU function. As mentioned above, the activation function includes the NPOT quantization scheme and the parameterized ReLU function. Typically, the system or the corresponding engine in the system can perform the operation corresponding to the specific activation function specified in formula (1). The system can enable or disable the NPOT quantization scheme and / or the parameterized ReLU function by adjusting the first and second coefficients of formula (1). In some implementations, the system can enable or disable the NPOT quantization scheme and / or the parameterized ReLU function using global parameters. The system can also derive other ReLU functions from formula (1) by adjusting the first and second coefficients.

[0101] Before processing the computation results using a specific activation function to generate the layer output, the system first determines whether to perform a bias operation on the computation results based on one or more criteria. These criteria include at least one of the following: whether the numerical distribution of the computation results is symmetric about a certain axis of symmetry, or whether the mean or median of the numerical distribution of the computation results is equal to zero. In response to determining to perform a bias operation on the computation results, the system generates a biased computation result by adding or subtracting a bias value from the computation results. The bias value is determined based on the numerical distribution of all possible computation results derived from the node inputs and corresponding node weights of the network layer.

[0102] The system further performs a pre-shift operation on the biased calculation result to generate an intermediate result. This intermediate result is stored in a data structure with a third precision, which is higher than the second precision but lower than the first precision. For example, the intermediate result can be stored using 12 bits (e.g., INT12), which has a precision higher than the second precision (8-bit data, e.g., INT8 or UINT8) but lower than the first precision (16-bit or 24-bit data, e.g., INT16 or INT24). Further details of the pre-shift operation are described above.

[0103] To perform the pre-shift operation, the system processes the biased calculation result according to a specific activation function expressed in formula (1), wherein the activation function contains the first coefficient defining the power value used in the NPOT quantization scheme. And the second coefficient of the parameterized ReLU function. First coefficient The second coefficient is greater than or equal to 1 and less than 2. Greater than 0 and less than or equal to 1.

[0104] The system further performs a pruning operation on the intermediate results. More specifically, the system determines whether the intermediate results meet pruning criteria. These criteria compare the resolution defined during quantization with the values ​​of the intermediate results. In response to determining that the intermediate results meet the pruning criteria, the system updates the intermediate results by performing a pruning operation, setting the values ​​of the intermediate results to one of the corresponding boundary values ​​defined during quantization. In some implementations, the system modifies the values ​​that meet the pruning criteria and adds a new value corresponding to one of the two boundary values ​​defining the resolution range. After performing the pruning operation, as described above, the system performs the operation specified by a specific activation function to generate node activation values.

[0105] After executing a specific activation function, the system performs a post-shift operation on the node activation values. By further reducing the data size and / or precision of the node activation values, candidate layer outputs are generated. This post-shift operation is similar to the pre-shift operation. The candidate layer output can be represented by INT8, whose data size is smaller than the data size representing the node activation values ​​(e.g., INT13).

[0106] The shift operation further includes a rounding operation. More specifically, the system performs a rounding operation on the least significant bit of the candidate layer output. If the least significant bit is greater than 0.5, the system rounds the candidate layer output up and prunes the rounded data to one of the two boundary values ​​of the data type.

[0107] The system can perform a similar pruning operation on the candidate layer output. More specifically, the system determines whether the candidate layer output meets the pruning criteria. The pruning criteria compare the resolution defined during quantization with the value of the candidate layer output (or the rounded-up candidate layer output). In response to determining that the candidate layer output meets the pruning criteria, the system updates the candidate layer output by performing a pruning operation, setting the value of the candidate layer output to one of the corresponding boundary values ​​defined during quantization.

[0108] The system further performs a truncation operation. More specifically, the system determines that the candidate layer output is greater than a threshold, and generates the layer output by setting the candidate layer output to the threshold. As mentioned above, the threshold is determined by formula (1).

[0109] In some implementations, the system may perform an add-shift operation on the output generated by one or more addition channels of a network layer. More specifically, the system receives another calculation result of the multiplication operation between the node input of another network layer of the neural network and the corresponding node weights of that other network layer, the other calculation result having a fourth precision different from the first precision. For example, the other calculation result may be stored in 8 bits (e.g., INT8). The system performs an add-shift operation on the other calculation result to align the decimal point of the other calculation result with the decimal point of the calculation result generated by the current layer. After decimal point alignment, the system 100 adds the outputs of the other network layer and the current network layer, and the result of the addition is further processed by the system using an operation defined by a specific activation function.

[0110] The system can store corresponding instruction data in one or more registers to process the calculation results using a specific activation function and generate layer output. For example, the one or more registers contain at least 24 bits. The corresponding instruction data may include: pre-shift operation instruction data stored in 4 bits, plus shift operation instruction data stored in 2 bits, NPOT quantization instruction data stored in 7 bits, parameterized ReLU function instruction data stored in 8 bits, and back shift operation instruction data stored in 3 bits.

[0111] The system generates neural network outputs (630) corresponding to the network inputs based on the layer outputs. For example, the network outputs may include predicted text sequences, audio sequences, image sequences, or other data used to perceive traffic conditions. The network outputs can be further used to generate control commands to control the operation of autonomous vehicles on the road.

[0112] In this specification, the term "machine learning model" refers to any model suitable for machine learning. For example, a machine learning model may include one or more neural networks trained to perform different inference tasks. Examples of neural networks and the tasks they perform are described in more detail at the end of this specification. For simplicity, the term "machine learning model" may sometimes be abbreviated as "neural network model" or "deep neural network" in this specification.

[0113] Depending on the task requirements, the neural network can be configured, for example, through training, to receive any type of digital data input and generate any type of score, classification result, or regression output based on the input.

[0114] In some cases, the neural network is a neural network configured to perform image processing tasks, namely, receiving an input image and processing the input image to generate a network output corresponding to the input image. In this specification, processing the input image refers to using a neural network to process the intensity values ​​of image pixels. For example, the task could be image classification, where the output generated by the neural network for a given image could be scores for a set of object categories, each score representing an estimated probability that the image contains an image of an object belonging to that category. Another example is image embedding generation, where the output generated by the neural network could be a numerical embedding of the input image. Yet another example is object detection, where the output generated by the neural network could identify the location of objects depicting a specific type in the input image. Yet another example is image segmentation, where the output generated by the neural network could assign each pixel of the input image to a category from a set of categories.

[0115] For example, if the input to a neural network is a sequence of texts in one language, the output generated by the neural network can be a set of scores for a set of text segments in another language, each score representing an estimated probability that the text segment in the other language is an appropriate translation of the input text into that language.

[0116] In some cases, the machine learning task is a combination of multiple individual machine learning tasks. For example, the neural network may be configured to perform multiple different individual machine learning tasks, such as two or more of the aforementioned machine learning tasks. For instance, the neural network may be configured to perform multiple individual image processing or computer vision tasks, such as generating the outputs of multiple different individual image processing tasks in parallel by processing a single input image.

[0117] Embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed herein and their structural equivalents, or combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, such as one or more computer program instruction modules encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access or serial access storage device, or combinations thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus.

[0118] The term "data processing device" refers to data processing hardware, encompassing all devices, equipment, and machines used for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, the device may include, in addition to hardware, code that creates a computer program execution environment, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0119] A computer program, also known as a program, software, software application, application program, module, software module, script, or code, can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may correspond to, but does not necessarily correspond to, a file in a file system. A program may be stored as part of a file that contains other programs or data, such as one or more scripts stored in a markup language specification, a single file dedicated to said program, or multiple coordination files, such as files storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer, or on multiple computers located in one location or distributed across multiple locations and interconnected through a data communication network.

[0120] A system consisting of one or more computers configured to perform a specific operation or action means that software, firmware, hardware, or a combination thereof are installed on the system, which, when run, causes the system to perform the operation or action. One or more computer programs configured to perform a specific operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0121] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from the inputs. An engine can be a coded functional block, such as a library, platform, software development kit (SDK), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, personal digital assistant, smartphone, or other fixed or portable device containing one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0122] The processes and logic described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic can also be executed by dedicated logic circuits such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmable computers.

[0123] A computer suitable for executing computer programs may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the CPU for executing or running instructions, and one or more storage devices for storing instructions and data. The CPU and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices for storing data, or operatively coupled to receive data from or transfer data to said mass storage device, or both, such as a magnetic disk, magneto-optical disk, or optical disk. However, a computer does not necessarily need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0124] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices, such as EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM (Compact Disc Read-Only Memory) and DVD-ROM (Digital Versatile Disc Read-Only Memory) optical disks.

[0125] To enable interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor, for displaying information to the user, and the computer also having a keyboard and pointing device, such as a mouse, trackball, or touch-sensitive display or other surface, through which the user provides input to the computer. Other types of devices can also be used to enable interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, sending a webpage to the web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device running a messaging application, such as a smartphone, and receiving response messages from the user.

[0126] Embodiments of the subject matter described in this specification can be implemented in computing systems that include backend components such as data servers, or middleware components such as application servers, or frontend components such as client computers having a graphical user interface, web browser, or application through which users can interact with the implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0127] The computing system may include clients and servers. Clients and servers are typically remotely separated and usually interact via a communication network. The client-server relationship is based on computer programs running on respective computers that have a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML (HyperText Markup Language) pages, to a user device, for example, to display data to a user interacting with the device and to receive user input; the device acts as a client. Data generated on the user device, such as the result of user interaction, can be received by the server from the device.

[0128] In addition to the above embodiments, the following embodiments are also innovative:

[0129] Example 1 is a method for performing an operation represented by a neural network comprising multiple neural network layers. The method includes: receiving a calculation result of a multiplication operation between node inputs of the network layers of the neural network and the weights of corresponding nodes of the network layers, wherein the calculation result is of a first precision; processing the calculation result with a specific activation function to generate a layer output, wherein the layer output is of a second precision lower than the first precision, the specific activation function being determined based on a quantization process using values ​​that are not multiples of 2, and the specific activation function generating a non-zero output when the calculation result is non-zero; and generating a neural network output corresponding to the network input based on the layer output.

[0130] Example 2 is the method of Example 1, which uses a specific activation function to process the calculation result to generate the layer output, including: performing a bias operation to generate a biased calculation result by adding or subtracting a bias value to the calculation result, wherein the bias value is determined based on the distribution of all possible calculation results derived from the node inputs and corresponding node weights of the network layer.

[0131] Example 3 is the method of Example 2, which uses a specific activation function to process the calculation result to generate layer output, including: performing a pre-shift operation on the biased calculation result to generate an intermediate result, wherein the intermediate result has a third precision that is higher than the second precision and lower than the first precision.

[0132] Example 4 is the method of Example 3, further comprising: processing the intermediate result with the specific activation function to generate node activation values, wherein the specific activation function includes a first coefficient associated with a power value used in the quantization process and a second coefficient associated with a parameterized modified linear unit (ReLU) function.

[0133] Example 5 is the method of Example 4, wherein the first coefficient is greater than or equal to 1 and less than 2, and the second coefficient is greater than 0 and less than or equal to 1.

[0134] Example 6 is a method of Example 4 or 5, and further includes performing a post-shift operation on the node activation value to generate a candidate layer output.

[0135] Example 7 is the method of Example 6, further comprising: determining that the intermediate result or candidate layer output meets the pruning criterion; in response to determining that the intermediate result meets the pruning criterion, updating the intermediate result by performing a pruning operation, setting the value of the intermediate result to one of the corresponding boundary values ​​defined in the quantization process; and in response to determining that the candidate layer output meets the pruning criterion, updating the candidate layer output by performing the pruning operation, setting the value of the candidate layer output to one of the corresponding boundary values ​​defined in the quantization process.

[0136] Example 8 is a method of Example 6 or 7, and further includes: performing a rounding operation on the least significant bit of the candidate layer output.

[0137] Example 9 is a method of any one of Examples 6 to 8, further comprising: determining that the candidate layer output is greater than a threshold; and generating the layer output by setting the candidate layer output to the threshold.

[0138] Example 10 is a method of any one of Examples 3 to 9, further comprising: receiving another calculation result from another network layer of the neural network, wherein the other calculation result is calculated by multiplying the node input of the other network layer of the neural network with the corresponding node weight of the other network layer, and the other calculation result has a fourth precision different from the first precision; and performing a shift operation on the other calculation result to align the decimal point of the other calculation result with the decimal point of the intermediate result.

[0139] Example 11 is a method of any one of Examples 1 to 10, wherein the second precision is represented by an 8-bit integer.

[0140] Example 12 is a method of any one of Examples 1 to 11, further comprising: storing corresponding instruction data in one or more registers for processing the calculation result using the specific activation function to generate the layer output, wherein the one or more registers contain at least 24 bits, and the corresponding instruction data includes (i) pre-shift operation instruction data stored in 4 bits, (ii) plus shift operation instruction data stored in 2 bits, (iii) quantization process instruction data stored in 7 bits; (iv) parameterized ReLU function instruction data stored in 8 bits, and back shift operation instruction data stored in 3 bits.

[0141] Example 13 is a system comprising one or more computers and a storage device for storing one or more instructions, wherein when executed by the one or more computers, the instructions cause the one or more computers to perform corresponding operations, the operations including the methods of any one of Examples 1 to 12.

[0142] Example 14 is one or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform a corresponding operation, the corresponding operation including the method of any one of Examples 1 to 12.

[0143] Although this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of claims, but rather as descriptions of specific features of specific embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in a particular combination, or even initially claimed in this manner, in some cases one or more features may be removed from the claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0144] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring that such operations must be performed in the specific order or sequence shown, or that all the operations shown must be performed to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments. Rather, it should be understood that the described program components and systems can generally be integrated into a single software product or packaged as multiple software products.

[0145] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the flow depicted in the figures does not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for performing an operation represented by a neural network comprising a plurality of neural network layers, the method comprising: The method includes: The calculation result of the multiplication operation between the node input of the network layer of the neural network and the corresponding node weight of the network layer is received, wherein the calculation result has a first precision; The computation result is processed using a specific activation function to generate a layer output, wherein the layer output has a second precision lower than the first precision, the specific activation function is determined based on a quantization process using values ​​that are not powers of 2, and the specific activation function generates a non-zero output when the computation result is non-zero; and The neural network output corresponding to the network input is generated based on the output of the layer.

2. The method of claim 1, wherein, Processing the computation results using a specific activation function to generate the layer output includes: A bias operation is performed to generate a biased calculation result by adding or subtracting a bias value to the calculation result, wherein the bias value is determined based on the distribution of all possible calculation results derived from the node inputs and corresponding node weights of the network layer.

3. The method according to claim 2, characterized in that, Processing the computation results using a specific activation function to generate the layer output includes: A pre-shift operation is performed on the biased calculation result to generate an intermediate result, wherein the intermediate result has a third precision that is higher than the second precision and lower than the first precision.

4. The method according to claim 3, characterized in that, Also includes: The intermediate results are processed using the specific activation function to generate node activation values, wherein the specific activation function includes a first coefficient associated with the power value used in the quantization process and a second coefficient associated with the parameterized modified linear unit (ReLU) function.

5. The method according to claim 4, characterized in that, The first coefficient is greater than or equal to 1 and less than 2, and the second coefficient is greater than 0 and less than or equal to 1.

6. The method according to claim 4, characterized in that, Also includes: A post-shift operation is performed on the node activation value to generate the candidate layer output.

7. The method according to claim 6, characterized in that, Also includes: Determine that the intermediate result or candidate layer output meets the pruning criteria; In response to determining that the intermediate result meets the clipping criteria, the intermediate result is updated by performing a clipping operation, and the value of the intermediate result is set to one of the corresponding boundary values ​​defined in the quantization process; as well as In response to determining that the candidate layer output meets the pruning criteria, the candidate layer output is updated by performing the pruning operation, and the value of the candidate layer output is set to one of the corresponding boundary values ​​defined in the quantization process.

8. The method according to claim 6, characterized in that, Also includes: Perform a rounding operation on the least significant bit of the candidate layer output.

9. The method according to claim 6, characterized in that, Also includes: The candidate layer output is determined to be greater than the threshold. as well as The layer output is generated by setting the candidate layer output to the threshold.

10. The method according to claim 3, characterized in that, Also includes: Receive another calculation result from another network layer of the neural network, wherein the other calculation result is calculated by multiplying the node input of the other network layer of the neural network with the corresponding node weight of the other network layer, and the other calculation result has a fourth precision different from the first precision; as well as Perform a shift operation on the other calculation result to align the decimal point of the other calculation result with the decimal point of the intermediate result.

11. The method according to claim 1, characterized in that, The second precision is represented by an 8-bit integer.

12. The method according to claim 1, characterized in that, Also includes: The corresponding instruction data is stored in one or more registers for processing the calculation result using the specific activation function to generate the layer output, wherein the one or more registers contain at least 24 bits, and the corresponding instruction data includes: (i) pre-shift operation instruction data stored in 4 bits, (ii) add shift operation instruction data stored in 2 bits, (iii) quantization process instruction data stored in 7 bits, (iv) parameterized ReLU function instruction data stored in 8 bits, and back shift operation instruction data stored in 3 bits.

13. A system comprising one or more computers and one or more storage devices for storing instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform corresponding operations, characterized in that... The operation includes: The calculation result of a multiplication operation between the node inputs of a network layer of a neural network comprising multiple neural network layers and the corresponding node weights of the network layer is received, wherein the calculation result has a first precision; The computation result is processed using a specific activation function to generate a layer output, wherein the layer output has a second precision lower than the first precision, the specific activation function is determined based on a quantization process using values ​​that are not powers of 2, and the specific activation function generates a non-zero output when the computation result is non-zero; and The neural network output corresponding to the network input is generated based on the output of the layer.

14. The system according to claim 13, characterized in that, Processing the computation results using a specific activation function to generate the layer output includes: A bias operation is performed to generate a biased calculation result by adding or subtracting a bias value to the calculation result, wherein the bias value is determined based on the distribution of all possible calculation results derived from the node inputs and corresponding node weights of the network layer.

15. The system according to claim 14, characterized in that, Processing the computation results using a specific activation function to generate the layer output includes: A pre-shift operation is performed on the biased calculation result to generate an intermediate result, wherein the intermediate result has a third precision that is higher than the second precision and lower than the first precision.

16. The system according to claim 15, characterized in that, The operation also includes: The intermediate results are processed using the specific activation function to generate node activation values, wherein the specific activation function includes a first coefficient associated with the power value used in the quantization process and a second coefficient associated with the parameterized modified linear unit (ReLU) function.

17. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations, characterized in that... The corresponding operations include: The calculation result of a multiplication operation between the node inputs of a network layer of a neural network comprising multiple neural network layers and the corresponding node weights of the network layer is received, wherein the calculation result has a first precision; The computation result is processed using a specific activation function to generate a layer output, wherein the layer output has a second precision lower than the first precision, the specific activation function is determined based on a quantization process using values ​​that are not powers of 2, and the specific activation function generates a non-zero output when the computation result is non-zero; and The neural network output corresponding to the network input is generated based on the output of the layer.

18. One or more computer-readable storage media according to claim 17, characterized in that, Processing the computation results using a specific activation function to generate the layer output includes: A bias operation is performed to generate a biased calculation result by adding or subtracting a bias value to the calculation result, wherein the bias value is determined based on the distribution of all possible calculation results derived from the node inputs and corresponding node weights of the network layer.

19. One or more computer-readable storage media according to claim 18, characterized in that, Processing the computation results using a specific activation function to generate the layer output includes: A pre-shift operation is performed on the biased calculation result to generate an intermediate result, wherein the intermediate result has a third precision that is higher than the second precision and lower than the first precision.

20. One or more computer-readable storage media according to claim 19, characterized in that, The operation also includes: The intermediate results are processed using the specific activation function to generate node activation values, wherein the specific activation function includes a first coefficient associated with the power value used in the quantization process and a second coefficient associated with the parameterized modified linear unit (ReLU) function.

Images