Optimizing low-precision inference models for deployment of deep neural networks

Abstract

This disclosure relates to optimizing low-precision inference models for deployment in deep neural networks. Systems, apparatus, and methods provide techniques for performing asymmetric quantization for optimizing inference neural network models, achieved by: generating a quantized neural network, wherein the model weights of the neural network are quantized to signed integer values, and wherein the input layer of the neural network is configured to quantize input values ​​to unsigned integer values; generating a weight accumulation table based on the quantized model weights and the kernel size of the neural network; and generating an output recovery function for the output layer of the neural network based on the weight accumulation table and the kernel size. The technique also performs per-input channel quantization. The technique also performs mixed-precision autotuning.

Description

Technical Field

[0001] The embodiments generally relate to machine learning. More specifically, the embodiments relate to optimizing neural network techniques for deployment in industrial, commercial, and consumer applications. Background Technology

[0002] Recent advancements in machine learning (ML) techniques, particularly neural networks, have shown promise for applications across a wide range of computational tasks. Neural networks, such as deep neural networks, can involve complex matrix-based multiplication and convolution operations. Once trained, a neural network can be deployed as an inference neural network model. However, the high computational complexity of deep neural networks (and other neural networks) presents challenges for deploying inference models in industrial, commercial, and / or consumer applications. Low-precision inference models have been considered, but lack adequate accuracy and / or have excessive memory or bandwidth requirements. Attached Figure Description

[0003] By reading the following specification and appended claims, and by referring to the following drawings, various advantages of the embodiments will become apparent to those skilled in the art, in which: Figure 1 A diagram is provided illustrating a system for optimizing an inference neural network model according to one or more embodiments; Figure 2A-2C Graphs of aspects of asymmetric quantization according to one or more embodiments are provided; Figures 3A-3B A graph of each input channel quantization aspect according to one or more embodiments is provided; Figure 4 A diagram is provided illustrating a system for optimizing an inference neural network model according to one or more embodiments; Figure 5 A flowchart illustrating a process for tuning an inference neural network model according to one or more embodiments is provided; Figures 6A-6B A diagram is provided illustrating a system for optimizing an inference neural network model according to one or more embodiments; Figure 7 A flowchart illustrating a process for optimizing an inference neural network model according to one or more embodiments is provided; Figure 8 A block diagram illustrating an example system for optimizing an inference neural network model according to one or more embodiments is provided; Figure 9 This is a block diagram illustrating an example semiconductor device for optimizing an inference neural network model according to one or more embodiments; Figure 10 This is a block diagram illustrating an example processor according to one or more embodiments; and Figure 11 This is a block diagram illustrating an example of a multiprocessor-based computing system according to one or more embodiments. Detailed Implementation

[0004] Applications such as image recognition and natural language processing (NLP) can utilize deep learning techniques, a subset of artificial intelligence (AI) machine learning, where neural networks such as deep neural networks (DNNs) contain multiple intermediate layers to perform complex operations on the input data. Because the amount of data involved in deep neural networks is relatively large, the data is often organized and processed as n-dimensional arrays (e.g., tensors), which can be further divided into matrices. Common matrix operations in this context can include matrix multiplication (e.g., "matmul" via a general matrix multiplication / GEMM kernel), convolution operations (e.g., via a convolution kernel), and so on. Inference neural network models can include low-precision quantization, as described below, and provide appropriately optimized performance for deployment.

[0005] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 1 A block diagram of an example system 110 for optimizing an inference neural network model using asymmetric quantization is shown according to one or more embodiments. System 110 may include a trained neural network model 112, an asymmetric quantization module 114, and an inference neural network engine 116. The trained neural network model 112 may be a deep neural network and may be trained according to neural network training procedures known to those skilled in the art. The trained neural network model 112 typically includes an input layer or stage for processing input tensors or other input data, including intermediate layers for weights of computational operations such as convolution operations or matrix multiplication operations, and an output layer or stage for processing or presenting output data.

[0006] The asymmetric quantization module 114 can perform one or more processes to optimize an inference neural network model based on a trained neural network model 112, such as an inference model implemented in an inference neural network engine 116. The inference neural network engine 116 can receive input tensors or other input data 132, perform computations or other operations based on the inference model (based on the trained model), and provide output tensors or other output data 136. The inference neural network engine 116 may include input layer quantization 122, model weight quantization 124, and output layer recovery 126. The input tensors or other input data 132 can be received as relatively high-precision values ​​(e.g., floating-point values ​​or high-precision integer values). Similarly, the output tensors or other output data 136 can be relatively high-precision values ​​(e.g., floating-point values ​​or high-precision integer values). Floating-point values ​​can be, for example, 64-bit floating-point values ​​(fp64) or 32-bit floating-point values ​​(fp32); high-precision integer values ​​can be, for example, 64-bit or 32-bit integer values ​​(int64 or int32).

[0007] The input tensor or other input data 132, received as relatively high-precision values, can be quantized by the input layer quantization 122 into low-precision integer values, such as 8-bit integer values ​​(int8). The inference model can perform computations or other operations, such as matrix multiplication and / or convolution operations, where the weights of the applicable kernel (e.g., GEMM or convolutional kernel) have been quantized into low-precision weights, such as 8-bit integer values ​​(int8), via model weight quantization 124. The output layer recovery 126 process converts the low-precision integer output values ​​back to high-precision (e.g., floating-point) output values.

[0008] The asymmetric quantization module 114 controls how input layer quantization 122, model weight quantization 124, and output layer recovery 126 are implemented or performed. Input layer quantization 122 can be achieved by using... asymmetry Quantization is achieved by quantizing the input values ​​so that each high-precision input value (e.g., fp32) is quantized according to the following formula: Unsigned Integer values ​​(e.g., uint8): in It is the quantized integer input value. It is the input scaling factor. It is a floating-point input value, and z This is either a bias or offset. Rounding functions can be applied to round up or down to the nearest integer value. For example, it can be based on the input value. Dynamic range, setting the input scaling factor and bias z This ensures that the quantized uint8 value fits within the range of 0 to 255. in and These are the minimum and maximum input (floating-point) values, respectively. In some embodiments, the input scaling factor can be set according to other criteria. and bias z .

[0009] Model weight quantization 124 can be used symmetry Quantization is achieved by quantizing floating-point weights so that each high-precision weight value (e.g., fp32) is quantized according to the following formula. Symbols Integer values ​​(e.g., int8): .

[0010] in These are the quantized integer weight values. It is the weight scaling factor, and These are floating-point weight values. Rounding functions can be applied to round up or down to the nearest integer value. For example, you can base the weight value... Dynamic range setting weight scaling factor This ensures that the quantized int8 values ​​fit within the range of -128 to +127. Model weight quantization of 124 can be performed once when the inference model is built.

[0011] The output value can be calculated based on the inference model. For example, the inference model can be convolved using a convolution kernel W according to the following formula: in The integer output value is (e.g., a 32-bit integer), and conv(x, W) represents a convolution operation involving the input value x and the kernel weights W. As an example, the input value x can represent a two-dimensional (2D) data set (e.g., an image), and the kernel W can represent a two-dimensional (2D) set of weights.

[0012] See below for reference Figure 2A-2C Further details related to some aspects of asymmetric quantization are illustrated and described. Refer to the components and features described herein (including, but not limited to, the accompanying figures and associated descriptions). Figure 2A The diagram illustrates padding used in convolution operations according to one or more embodiments. For convolution operations, the input data set may be padded with values ​​(e.g., around the outer boundary) to accommodate convolution operations involving kernels with input values ​​along the outer boundary. Figure 2AA two-dimensional input data set 212 is depicted, which can represent, for example, an image. For illustration, data set 212 is shown as a 16 x 16 set of data values ​​(e.g., pixels), but the size of the data set can be any size (and the sizes do not need to be equal, but can be, for example, 32 x 64). Box 214 shows the outline of the kernel that can be used in the convolution operation; for illustration, the kernel size is shown as a 3 x 3 kernel, but the kernel can be any size (and the kernel size does not need to be equal, but can be, for example, 5 x 7).

[0013] like Figure 2A As shown, when the kernel is centered on the top-left data value, some kernel values ​​do not overlap with any input values. Therefore, these values ​​are added as "padding" for the convolution operation, such as... Figure 2A As shown, this figure depicts two-dimensional input data with padding 216. The padding data set 216 is displayed as a series of additional data elements (e.g., pixels), arranged in rows and columns around the outer boundary of the data set 212. The number of padding rows and columns depends on the kernel size. For the example of a 3x3 kernel, a single padding row is added to the top and bottom of the two-dimensional data set 212, and a single padding column is added to each side of the two-dimensional data set 212. The padding value can be set to zero, or it can be set to a constant value, and so on. For inference engines as described in this article, the padding value can typically be set to zero. An example of how a 3x3 kernel can cover the padding data set 216 is shown in [the image / description]. Figure 2A The input is represented as a series of boxes 218a-218i. More generally, the number of rows or columns to be added as padding along each boundary of the input is equal to trunc(K / 2), where K is the kernel size and trunc() denotes the truncation of the result (e.g., K=3 for a 3x3 kernel).

[0014] Once all convolutions and / or other computational operations of the inference model are complete, the output value can be converted (i.e., dequantized) to recover a high-precision (e.g., floating-point) value. Output layer recovery 126, via asymmetric quantization module 114, can be implemented as a recovery function by converting the integer output value to a high-precision output value (e.g., fp32) according to the following formula: in, It is a high-precision floating-point output value. It is an integer value output from the inference model. and These are the input and weight scaling factors, respectively. It is a weighted cumulative table. Defined as: in It is an applicable kernel index. and A set of integer kernel weights within a certain range.

[0015] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 2B The figure illustrates a weight accumulation table according to one or more embodiments. )222. As shown in the figure, as an example, the weight accumulation table 222 corresponds to a 3 x 3 kernel, unit span, where single-row padding is added around the boundary rows / columns of the input dataset (e.g. Figure 2A (As shown in the padding data set 216). Unit span refers to the kernel being shifted by a single index value each time during convolution.

[0016] like Figure 2B As shown, The top-left element of 222, wacc[0,0], corresponds to Figure 2A The kernel is covered by box 218a. Move to the right. The next element wacc[0,1] of 222 corresponds to the kernel covered by box 218b, and The next element wacc[0,2] of 222 corresponds to the kernel covered by box 218c. Similarly, Element wacc[1,0] of 222 corresponds to the kernel covered by box 218d, element wacc[1,1] corresponds to the kernel covered by box 218e, element wacc[1,2] corresponds to the kernel covered by box 218f, and is moved to the bottom of the weight accumulation table. Element wacc[2,0] corresponds to the kernel covered by box 218g, element wacc[2,1] corresponds to the kernel covered by box 218h, and element wacc[2,2] corresponds to the kernel covered by box 218i. This will be understood. The values ​​wacc[0,0], wacc[0,2], wacc[2,0], and wacc[2,2] each correspond to a single point in the convolution, where the kernel is centered around one of the input values ​​at each corner. It will also be understood that... The values ​​wacc[0,1] correspond to points in the convolution where the kernel is centered on the top row of the input values ​​(excluding corner values), wacc[2,1] corresponds to points in the convolution where the kernel is centered on the bottom row of the input values ​​(excluding corner values), wacc[1,0] corresponds to points in the convolution where the kernel is centered on the leftmost column of the input values ​​(excluding corner values), and wacc[1,2] corresponds to points in the convolution where the kernel is centered on the rightmost column of the input values ​​(excluding corner values). Finally, it will also be understood that... The value wacc[1,1] corresponds to a point in the convolution where the kernel is centered at any input value except the boundary values. The coordinates of the output layer can correspond to the coordinates of the input layer (i.e., the input dataset) without padding. A mapping from the output layer's output coordinates to the indices of the weight accumulation table can be generated based on the kernel size, span, and size of the output dataset (e.g., dimension) to reflect the application of the weight accumulation table values ​​to the coordinates of the dataset as described above, enabling rapid computation of high-precision output values ​​according to Equation 5. Table 224 shows the arrangement of the values ​​in the weight accumulation table 222, where the positions of individual table elements roughly correspond to the mappings to the respective output coordinates. Using the above reference... Figure 2B The described weighted cumulative table allows for the efficient ignoring of calculations involving filling data elements, thereby further improving the efficiency and speed of computation.

[0017] According to the formula (6) above, each entry in the weighted cumulative table 222 is calculated as compared with the entry shown in gray. The sum of the individual kernel weights corresponding to the individual boxes of each element in 222. For example, wacc[0,0] is the sum of the 4 kernel weights displayed in gray, wacc[0,1] is the sum of the 6 kernel weights displayed in gray, wacc[1,1] is the sum of all 9 kernel weights displayed in gray, and so on. Figure 2C An example of a weight accumulation table 234 for a sample 3x3 kernel 232 with weights ranging from 1 to 5 is illustrated, and the corresponding values ​​in the weight accumulation table 234 are calculated based on the values ​​of the kernel 232. For example, the value wacc[1,1] is the sum of all weights in the kernel, which is 17 for the sample kernel 232, as shown in the corresponding weight accumulation table 234 (the center value is wacc[1,1]).

[0018] As referenced above Figure 2A-2CAs the diagram and examples illustrate, a 3 x 3 kernel size results in a weight accumulation table with 9 elements. For example, with a kernel size of 5 x 5, the corresponding weight accumulation table would include 25 elements. In cases where the span is longer than a unit span, the mapping between the weight accumulation table values ​​and the indexes from the output coordinates to the weight accumulation table can be adjusted accordingly.

[0019] In some embodiments, the inference model may have multiple internal levels, such as multiple convolutional levels, with a different kernel used for each level. In this case, each level may have a separate per-level weight accumulator table, and an aggregated weight accumulator table (which will be used to compute the output value) can be constructed by adding the individual elements of each per-level weight accumulator table. For example, the elements of the aggregated weight accumulator table may be computed as follows: (6B) in, Let L be the per-level weight cumulative table for level n, where L is the total number of internal levels, and (i,j) is the index of each per-level weight cumulative table. Each per-level weight cumulative table can be calculated, for example, as described in Equation 6A above. .

[0020] More generally, when the input is a tensor with multiple channels, the corresponding weight accumulator will have an additional dimension. This means the weight accumulator will have a set of entries extending into the additional dimension, with one level corresponding to each channel. For example, if the input tensor is an image with color values ​​and 3 channels, such as red, green, and blue, and if the kernel size is 5x5 (unit span), the corresponding weight accumulator will be a 5x5x3 matrix. This means the weight accumulator will have 3 sets (or levels) of 5x5 entries in the additional matrix dimension (here, the third dimension), with one set (or level) of 5x5 entries corresponding to each channel.

[0021] In cases where the input is a multi-channel tensor, in some embodiments, additional optimizations to the inference engine can be performed on a per-channel basis, taking into account the relative dynamic range of the input value for each channel. Figure 3A The diagram shown depicts the range of values ​​for each input channel of an example tensor with two input channels, C1 (label 312) and C2 (label 316). As illustrated in this example, the input values ​​for input channel C1 range from -8.0 (label 314) to +1.0 (label 315), and the input values ​​for input channel C2 range from -1.0 (label 318) to +1.0 (label 319). (If using the reference above...) Figure 1 , Figure 2A-2CThe described input quantization applies the same scaling factor and bias to both channels, potentially causing the quantization of one channel to be compressed or unbalanced relative to the other. However, per-input-channel quantization can be performed to account for the dynamic range of each channel, where quantization is computed individually for each input channel, resulting in improved accuracy.

[0022] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 4 A block diagram of an example system 410 for optimizing an inference neural network model using a combination of asymmetric quantization and per-input channel quantization is shown according to one or more embodiments. System 410 may include a trained neural network model 112 (as described above), an asymmetric quantization module 114 (as described above), a per-input channel quantization module 415, and an inference neural network engine 416. The per-input channel quantization module 415 may be combined with the asymmetric quantization module 114 to perform one or more processes for optimizing an inference neural network model based on the trained neural network model 112, such as an inference model implemented in the inference neural network engine 416. The inference neural network engine 416 may receive multi-channel input tensors or other multi-channel input data 432, perform computations or other operations according to the inference model (based on the trained model), and provide output tensors or other output data 436. The inference neural network engine 416 may include input layer quantization 422, model weight quantization 424, and output layer recovery 426. The input tensors or other input data 432 may be received as relatively high-precision values ​​(e.g., floating-point values ​​or high-precision integer values). Similarly, the output tensor or other output data 436 can be a relatively high-precision value (e.g., a floating-point value or a high-precision integer value). Floating-point values ​​can be, for example, 64-bit floating-point values ​​(fp64) or 32-bit floating-point values ​​(fp32); high-precision integer values ​​can be, for example, 64-bit or 32-bit integer values ​​(int64 or int32).

[0023] The input tensor or other input data 432, received as relatively high-precision values, can be quantized by the input layer quantization 422 on a per-channel basis to low-precision integer values, such as 8-bit integer values ​​(int8). The inference model can perform computations or other operations, such as matrix multiplication and / or convolution operations, where the weights of the applicable kernel (e.g., GEMM or convolutional kernel) have been quantized to low-precision weights, such as 8-bit integer values ​​(int8), via model weight quantization 424. The output layer recovery 426 process converts the low-precision integer output values ​​back to high-precision (e.g., floating-point) output values ​​(e.g., floating-point). The asymmetric quantization module 114, in conjunction with the per-input-channel quantization module 415, controls how input layer quantization 422, model weight quantization 424, and output layer recovery 426 are implemented or performed. Input layer quantization 422 can be achieved by using... asymmetry Quantization is achieved by quantizing the input value on a per-channel basis, so that each high-precision input value (e.g., fp32) is quantized according to the following formula: Unsigned Integer values ​​(e.g., uint8): (7) here It is the quantized integer input value of input channel c. It is the input scaling factor for input channel c. It is the floating-point input value of input channel c, and This is the bias or offset of the input channel c. A rounding function can be applied to round up or down to the nearest integer value. For each input channel c, a rounding function can be applied based on the input value. Use the dynamic range to set the input scaling factor and bias This ensures that the quantized uint8 value fits within the range of 0-255. (8) ;as well as in and These are the minimum and maximum input (floating-point) values ​​for input channel c, respectively. In some embodiments, the minimum and maximum input values ​​for each channel can be determined based on the training input data 420. and The input data can be data used to train the trained neural network model 112.

[0024] As an example, Figure 3B The above formulas (7) and (8) are shown to be used for... Figure 3A The input value reflected in the text is the quantized input value calculated on a per-channel basis. For channel 1 (C1), as shown... Figure 3A As shown, the minimum value of 314 is (-8.0), and the maximum value of 315 is +1.0. For channel C1 in... Figure 3B The scaling factor, bias, and quantization value are shown in Figure 322. Column Xfp (label 324) shows the floating-point input value, and column Xuint8 (label 328) shows the corresponding quantization value, ranging from 0 to 255. As shown in Figure 322, for channel C1, the input value 0.0 is quantized to 227. For channel 2 (C2), as... Figure 3A As shown, the minimum value of 318 is (-1.0), and the maximum value of 319 is +1.0. For channel C2... Figure 3B The scaling factor, bias, and quantization value are shown in Figure 332. Column Xfp (label 334) shows the floating-point input value, and column Xuint8 (label 338) shows the corresponding quantization value, ranging from 0 to 255. As shown in Figure 332, for channel C2, the input value 0.0 is quantized to 128.

[0025] Return to Figure 4 Model weight quantization 424 can be achieved by using symmetry Quantization is achieved by quantizing the floating-point weights on a per-channel basis, so that each high-precision weight value (e.g., fp32) is quantized according to the following formula. Symbols Integer values ​​(e.g., int8): (9) in These are the quantized integer weight values ​​of input channel c. It is the weight scaling factor for input channel c, and This is the floating-point weight value of input channel c. A rounding function can be applied to round it up or down to the nearest integer value. Per-channel weight scaling factor. It can be determined using the following formula: (10) in (11) Therefore, the weight scaling factor per channel The calculation method ensures that the result of the multiplication can be accumulated along the input channel, and the scaling factor of the result is... As shown in Equation 11, it is no different from the scaling factor based on the tensor. Smaller, therefore more ideal. Model weight quantization 424 can be performed once when the inference model is built.

[0026] As referenced above Figure 1 The output value can be calculated based on the inference model. For example, for each input channel, the inference model can perform a convolution operation via a convolution kernel W according to the following formula: (12) in (e.g., a 32-bit integer) is the integer output value, and conv(x, W) represents the convolution operation involving the input value x and the kernel weight W for channel c. Once all convolutions and / or other computational operations of the inference model are complete, the output value can be converted (i.e., dequantized) to recover a high-precision (e.g., floating-point) value. Output layer recovery 426 can be achieved by using the conversion of the integer output value to a high-precision output value (e.g., fp32) according to the following formula: (The provided text is incomplete and requires further context.) (13) in It is a high-precision floating-point output value. It is an integer value output from the inference model. and These are the input and weight scaling factors, respectively. It is a multidimensional weighted cumulative table with one level for each channel, as described above.

[0027] In some embodiments, additional optimization of the inference model can be achieved through mixed-precision autotuning. Refer to the components and features described herein (including, but not limited to, the accompanying figures and descriptions). Figure 5 A flowchart illustrating process 510 for tuning an inference neural network model according to one or more embodiments is provided. Hybrid precision auto-tuning can be applied to the inference engine 116 (…). Figure 1 ) or inference engine 416 ( Figure 4 The inference model is implemented in [the code]. The tuning process may include performing a series of test runs on the model to determine whether the model accuracy can be improved by changing the precision used in quantization. This process begins with the current state of the inference model (block 512). In block 514, the inference model is run and the accuracy of the result is determined. In block 516, the result is compared with an accuracy criterion. Accuracy criteria may include, for example, mean average precision (mAP). Other accuracy criteria may be used for the accuracy test in block 516, such as the optimal mean squared error (OMSE) procedure described below. If the accuracy of the result passes the accuracy criterion evaluation, the process continues at block 530 (as described below). If the accuracy of the result does not pass the accuracy criterion evaluation in block 516, the process continues to block 518.

[0028] In block 518, it is determined whether another precision for the inputs and / or weights is available. For example, int16 quantization can be chosen as an alternative to int8 quantization. As another example, restoring the precision to floating-point (e.g., fp32 or fp16) can be chosen as an alternative. If another precision is available in block 518, the process continues to block 520, where the precision of the inference model is adjusted to the alternative choice, and the process returns to block 514, where the model is run again and the adjusted precision is applied. If no other precision is available in block 518 (e.g., all possible alternative precisions have been tried), the process continues to block 522. In some embodiments, the determination of available precision (block 518) and the adjustment of precision (block 520) can be performed on an algorithm-by-algorithm basis for the inference model; in this case, once all available precision adjustments have been made for a particular algorithm, the process continues to block 522.

[0029] In block 522, it is determined whether all algorithms have been applied (e.g., for a specific layer of the inference model). If so, the process continues to block 524, where another layer is selected and its precision is adjusted. The process then returns to block 514, where the model is run again and the adjusted precision is applied. In some embodiments, the adjustment process can begin from the last layer, and once all algorithms for that layer have been applied, the process can “fall back” to previous layers, and the precision can be scheduled, for example, adjusted to fp32. If it is determined in block 522 that not all algorithms have been applied, the process continues to block 526.

[0030] In block 526, the tensor data can be analyzed and a new algorithm can be applied. The new algorithm can be kernel-based or an alternative algorithm to the neural network model. The process then returns to block 514, where the model is run again and the new algorithm is applied.

[0031] In block 530, the process has continued from block 516, where the accuracy criterion evaluation has been passed. Any adjustments made (e.g., the accuracy adjustments in block 520) can be incorporated into the updated inference model, which can be implemented in the inference engine, and the tuning process exits at 510.

[0032] Figures 6A-6B This illustration depicts an alternative system for optimizing inference neural network models, incorporating automatic adjustment of mixed precision. Refer to the components and features described herein (including, but not limited to, the accompanying figures and associated descriptions). Figure 6A A block diagram of an example system 610 for optimizing an inference neural network model using asymmetric quantization and autotuning is shown according to one or more embodiments. Figure 6A Including, for example Figure 1As shown and referenced above Figure 1 The described optimization system may include an automatic adjustment module 616. The automatic adjustment module 616 can perform the functions described in the reference above. Figure 5 The automatic adjustment process is described. The automatic adjustment module 616 can receive data from the inference engine 116 and can operate in conjunction with the asymmetric quantization module 114 to apply alternative accuracy, etc., when running tests and when applying model updates once the tests are completed.

[0033] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 6B A block diagram of an example system 620 for optimizing an inference neural network model by combining asymmetric quantization with per-input channel quantization and autotuning is shown according to one or more embodiments. Figure 6B Including, for example Figure 4 As shown and referenced above Figure 4 The described optimization system may include an automatic adjustment module 626. The automatic adjustment module 626 can perform the functions described in the above reference. Figure 5 The automatic adjustment process is described. The automatic adjustment module 626 can receive data from the inference engine 416 and can operate in conjunction with the asymmetric quantization module 414 and the per-input-channel quantization module 415 to apply alternative accuracy during test run and when applying model updates once the test is complete, etc.

[0034] Quantization metrics can be used to determine the performance of an inference model or inference engine, for example, by evaluating accuracy. For instance, quantization metrics can be included as part of the mixed-precision autotuning process described above. During the design phase (e.g., post-training), validation datasets can be used as part of the evaluation or testing of the inference neural network model. However, in production or deployment, validation datasets may not be usable as part of the inference model evaluation or testing. Therefore, another metric, Optimal Mean Squared Error (OMSE), can be used to evaluate or test the inference model in production or deployment, and can also be used during the design process. OMSE can be calculated to indicate the difference between integer quantization (e.g., signed / unsigned int8) and floating-point representation (e.g., fp32) as described above.

[0035] It can be assumed or understood that the data distribution follows a Laplace distribution, which is a typical normal distribution in neural networks such as DNNs. Let x It has a probability density function f(x) The tensor is a random variable with fp32 precision. Without loss of generality, it can be assumed or understood that a preprocessing step has been performed to make the mean value in the tensor zero, for example: .

[0036] For int8 quantization, tensor values ​​can be uniformly quantized to 256 discrete values ​​from 0 to 255. The optimal maximum value α of the fp32 tensor can be calculated using the quantization algorithm. For any pruning function It can be defined as follows.

[0037] (14) The quantization step size Δ between two adjacent quantized values ​​can be established as Δ = 2α / 256, and x and its quantized version Q(x) The OMSE between them can be determined using the following formula: (15) The OMSE representation was evaluated against mean average precision (mAP), and the results are listed in Table 1 below. For the OMSE evaluation, a DNN model, SSD-MobileNetV1 (from the MLPerf inference track, a performance benchmark), was used. Quantization was applied as described above, and the conditioning process described above was employed to provide a series of sample test runs. The results are summarized in Table 1 below: Table 1: OMSE and mAP as quantitative evaluation metrics Table 1 shows a set of numbers, with the following headings: # - Indicates the sample test run number (ranging from 1 to 30). mAP - Indicates the evaluation performed using the validation dataset according to the mAP standard. omse - indicates an evaluation performed according to the OMSE process described above, using calibrated images as the input data set.

[0038] A higher mAP% (score) indicates better performance, while a lower OMSE value (score) indicates even better performance. For example, test run 1 has an evaluation result of mAP=70.88% and OMSE=0.098; test run 2 has an evaluation result of mAP=71.65% and OMSE=0.094; and test run 3 has an evaluation result of mAP=71.09% and OMSE=0.099. The OMSE evaluation result roughly tracks the mAP result, so in many (though not all) cases, test runs with better OMSE performance also have better mAP performance. For example, test run 2 has the best performance in both mAP and OMSE metrics. Therefore, the results show that the OMSE evaluation process described above is an effective way to evaluate the performance of quantized inference neural network models.

[0039] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 7 This is a flowchart illustrating a method 710 for optimizing an inference neural network model according to one or more embodiments.

[0040] In block 712, a quantized neural network can be generated, wherein the model weights of the neural network can be quantized into signed integer values, and wherein the input layer of the neural network can be configured to quantize the input values ​​into unsigned integer values.

[0041] In block 714, a weight accumulation table can be generated based on the quantized model weights and the kernel size of the neural network.

[0042] In block 716, the output recovery function can be generated based on the weight accumulation table and the kernel size.

[0043] In some embodiments, in block 722, a mapping between the output coordinates of the output layer and the indexes of the weight accumulation table can be generated. See above for reference. Figure 2A-2C The mapping is determined as described.

[0044] In some embodiments, method 710 may additionally or alternatively implement the above reference. Figures 3A-3B and Figure 4 Some aspects of per-input-channel quantization are described. In block 732, the input values ​​can be quantized on a per-channel basis, which can be achieved by configuring the input layer of the neural network. In block 734, the model weights of the neural network can be quantized on a per-channel basis.

[0045] In some embodiments, at block 742, the system may additionally or alternatively perform an automatic adjustment process. This automatic adjustment process may include the above-referenced... Figure 5 and Figures 6A-6B The description refers to some or all aspects of the automatic adjustment of mixing precision.

[0046] The above references Figures 1-7 (include Figure 1 and Figure 7 The systems and methods described herein can be implemented in any number of ways, including hardware (e.g., a processor) that executes software instructions. As an example, in one or more embodiments, the above references... Figures 1-7 (include Figure 1 and Figure 7 The systems and methods described herein can be implemented via one or more Intel® Xeon® Scalable processors that execute software instructions. Intel ® Xeon ® Scalable processors may include those with Intel ®Hardware acceleration support for Deep Learning (DL) enhancements (including an extended instruction set for integer operations).

[0047] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 8 The diagram illustrates a block diagram of an example computing system 10 for optimizing an inference neural network model according to one or more embodiments. System 10 may generally be part of an electronic device / platform having computing and / or communication functions (e.g., a server, cloud infrastructure controller, database controller, laptop computer, desktop computer, personal digital assistant / PDA, tablet computer, convertible tablet device, smartphone, etc.), imaging functions (e.g., camera, camcorder), media playback functions (e.g., smart TV), wearable functions (e.g., watch, glasses, headwear, footwear, jewelry), vehicle functions (e.g., car, truck, motorcycle), robotic functions (e.g., autonomous robot), etc., or any combination thereof. In the illustrated example, system 10 may include a host processor 12 (e.g., a central processing unit / CPU) having an integrated memory controller (IMC) 14 that can be coupled to system memory 20. The host processor 12 may include any type of processing device, such as a microcontroller, microprocessor, RISC processor, ASIC, etc., and associated processing modules or circuitry. System memory 20 may include any non-transitory machine or computer-readable storage medium such as RAM, ROM, PROM, EEPROM, firmware, flash memory, configurable logic such as PLA, FPGA, CPLD, fixed-function hardware logic using circuit technologies such as ASIC, CMOS, or TTL, or any combination thereof suitable for storing instructions 28.

[0048] System 10 may also include an input / output (I / O) subsystem 16. I / O subsystem 16 may communicate with, for example, one or more input / output (I / O) devices 18, a network controller (e.g., a wired and / or wireless NIC), and a storage device 22. Storage device 22 may include any suitable non-transitory machine or computer-readable storage type (e.g., flash memory, DRAM, SRAM (Static Random Access Memory), solid-state drive (SSD), hard disk drive (HDD), optical disk, etc.). Storage device 22 may include a mass storage device. In some embodiments, host processor 12 and / or I / O subsystem 16 may communicate with storage device 22 (all or part thereof) via network controller 24. In some embodiments, system 10 may also include a graphics processor 26.

[0049] The host processor 12, I / O subsystem 16, and / or graphics processor 26 can execute program instructions 28 retrieved from system memory 20 and / or storage device 22 to perform one or more aspects of the above-described process, including those referenced herein. Figure 1 and Figure 2A-2C The process described in this paper for asymmetric quantization is referenced. Figures 3A-3B and Figure 4 The process described for per-input channel quantization, and references in this paper. Figure 5 and Figures 6A-6B The process described is for automatic adjustment of mixed precision. The host processor 12 and / or I / O subsystem 16 can execute program instructions 28 retrieved from system memory 20 and / or storage device 22 to perform the procedures described herein. Figure 7 One or more aspects of the process used to optimize an inference neural network model are described.

[0050] Computer program code for performing the processes described above can be written and implemented as program instructions 28 using any combination of one or more programming languages, including object-oriented programming languages ​​such as JAVA, JAVASCRIPT, PYTHON, SMALLTALK, C++, etc., and / or traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. Furthermore, program instructions 28 may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, status setting data, configuration data for integrated circuits, status information of individual electronic circuits and / or other hardware-native structural components (e.g., host processor, central processing unit / CPU, microcontroller, microprocessor, etc.).

[0051] The host processor 12 and I / O subsystem 16 may be implemented together on a semiconductor die as a system on chip (SoC) 11, shown in solid lines. The SoC 11 can therefore operate as a computing device for optimizing inference models. In some embodiments, the SoC 11 may also include one or more of system memory 20, network controller 24, and / or graphics processor 26 (shown in dashed lines).

[0052] I / O device 18 may include one or more input devices, such as a touchscreen, keyboard, mouse, cursor control device, microphone, digital camera, video recorder, camcorder, biometric scanner, and / or sensor; the input devices can be used to input information and interact with system 10 and / or other devices. I / O device 18 may also include one or more output devices, such as displays (e.g., touchscreen, liquid crystal display / LCD, light-emitting diode / LED display, plasma panel, etc.), speakers, and / or other visual or audio output devices. Input and / or output devices can be used, for example, to provide a user interface.

[0053] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 9 The diagram illustrates a block diagram of an example semiconductor device 30 for optimizing an inference neural network model according to one or more embodiments. The semiconductor device 30 may be implemented, for example, as a chip, die, or other semiconductor package. The semiconductor device 30 may include one or more substrates 32 made of, for example, silicon, sapphire, gallium arsenide, etc. The semiconductor device 30 may also include logic 34 composed of, for example, transistor arrays and other integrated circuit (IC) components coupled to the substrate(s) 32. The logic 34 may implement the above-referenced... Figure 8 The described system-on-chip (SoC) 11. Logic 34 can implement one or more aspects of the above process, including those referenced herein. Figure 1 and Figure 2A-2C The process described in this paper for asymmetric quantization is referenced. Figures 3A-3B and Figure 4 The process described for per-input channel quantization, and references in this paper. Figure 5 and Figures 6A-6B This describes the process for automatic adjustment of mixed precision. Logic 34 can implement the procedures described in this article. Figure 7 One or more aspects of the process used to optimize an inference neural network model are described.

[0054] Semiconductor device 30 can be constructed using any suitable semiconductor manufacturing process or technology. Logic 34 can be at least partially implemented in configurable logic or fixed-function hardware logic. For example, logic 34 can include transistor channel regions located (e.g., embedded) within substrate(s) 32. Thus, the interface between logic 34 and substrate(s) 32 may not be an abrupt junction. Logic 34 can also be considered as including an epitaxial layer grown on an initial wafer of substrate(s) 34.

[0055] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 10This is a block diagram illustrating an example processor core 40 according to one or more embodiments. Processor core 40 can be the core of any type of processor, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, or other device for executing code. Although in Figure 10 The image only shows one processor core 40, but the processing element may include more than one. Figure 10 The processor core 40 shown is a single-threaded core, or in at least one embodiment, the processor core 40 may be multi-threaded, as it may include more than one hardware thread context (or "logical processor") for each core.

[0056] Figure 10 The diagram also illustrates a memory 41 coupled to the processor core 40. The memory 41 may be any of various types of memory known to those skilled in the art or otherwise available, including various layers of a memory hierarchy. The memory 41 may include one or more code 42 instructions to be executed by the processor core 40. The code 42 may implement one or more aspects of the processes described above, including those referenced herein. Figure 1 and Figure 2A-2C The process described in this paper for asymmetric quantization is referenced. Figures 3A-3B and Figure 4 The process described for per-input channel quantization, and references in this paper. Figure 5 and Figures 6A-6B This describes the process for automatic adjustment of blending precision. Code 42 can be implemented as referenced in this article. Figure 7 This describes one or more aspects of the process for optimizing an inference neural network model. Processor core 40 follows a program sequence of instructions indicated by code 42. Each instruction can enter front-end section 43 and be processed by one or more decoders 44. Decoder 44 can generate micro-operations, such as fixed-width micro-operations of a predetermined format, as its output, or can generate other instructions, micro-instructions, or control signals reflecting the instructions in the original code. The illustrated front-end section 43 also includes register renaming logic 46 and scheduling logic 48, which generally allocate resources and queue operations corresponding to the translation instructions for execution.

[0057] Processor core 40 is shown to include execution logic 50 having a set of execution units 55-1 to 55-N. Some embodiments may include several execution units dedicated to a specific function or set of functions. Other embodiments may include only one execution unit or a single execution unit capable of performing a specific function. The illustrated execution logic 50 performs operations specified by code instructions.

[0058] After the execution of the operation specified by the code instructions is completed, the backend logic 58 retires the instructions of code 42. In one embodiment, processor core 40 allows out-of-order execution of instructions but requires ordered retirement of instructions. Retirement logic 59 may take various forms known to those skilled in the art (e.g., a reordering buffer). Thus, processor core 40 is transformed during the execution of code 42, at least in terms of the output generated by the decoder, the hardware registers and tables utilized by register renaming logic 46, and any registers (not shown) modified by execution logic 50.

[0059] Although Figure 10 Not illustrated, but the processing element may include other on-chip components along with the processor core 40. For example, the processing element may include memory control logic along with the processor core 40. The processing element may include I / O control logic and / or may include I / O control logic integrated with the memory control logic. The processing element may also include one or more caches.

[0060] Refer to the components and features described herein (including but not limited to the accompanying drawings and related descriptions). Figure 11 This is a block diagram illustrating an example of a multiprocessor-based computing system 60 according to one or more embodiments. The multiprocessor system 60 includes a first processing element 70 and a second processing element 80. Although two processing elements 70 and 80 are shown, it is to understand that embodiments of the system 60 may also include only one such processing element.

[0061] System 60 is illustrated as a point-to-point interconnect system, wherein a first processing element 70 and a second processing element 80 are coupled via a point-to-point interconnect 71. It should be understood that... Figure 11 Any or all of the interconnects shown can be implemented as multipoint branch buses, rather than point-to-point interconnects.

[0062] like Figure 11 As shown, each of the processing elements 70 and 80 can be a multi-core processor, including first and second processor cores (i.e., processor cores 74a and 74b and processor cores 84a and 84b). These cores 74a, 74b, 84a, and 84b can be configured to... (The sentence is incomplete and requires further context). Figure 10 The instruction code is executed in a similar manner.

[0063] Each processing element 70, 80 may include at least one shared cache 99a, 99b. The shared cache 99a, 99b may store data (e.g., instructions) utilized by one or more components of the processor, such as cores 74a, 74b and 84a, 84b, respectively. For example, the shared cache 99a, 99b may store data in local cache memories 62, 63 for faster access by the processor components. In one or more embodiments, the shared cache 99a, 99b may include one or more intermediate level caches, such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, last level cache (LLC), and / or combinations thereof.

[0064] Although shown as having only two processing elements 70, 80, it is to be understood that the scope of the embodiments is not limited thereto. In other embodiments, one or more additional processing elements may be present in a given processor. Alternatively, one or more of the processing elements 70, 80 may be elements other than a processor, such as an accelerator or a field-programmable gate array (FPGA). For example, the additional processing elements(s) may include one or more additional processors identical to the first processor 70, one or more additional processors heterogeneous or asymmetric to the first processor 70, accelerators (e.g., graphics accelerators or digital signal processing (DSP) units), FPGAs, or any other processing element. There may be a variety of differences between the processing elements 70, 80 in terms of the range of value metrics including architectural characteristics, microarchitectural characteristics, thermal characteristics, power consumption characteristics, etc. These differences may actually manifest themselves as asymmetry and heterogeneity between the processing elements 70, 80. For at least one embodiment, the various processing elements 70, 80 may be present in the same die package.

[0065] The first processing element 70 may further include a memory controller logic (MC) 72 and point-to-point (PP) interfaces 76 and 78. Similarly, the second processing element 80 may include an MC 82 and PP interfaces 86 and 88. Figure 11 As shown, MCs 72 and 82 couple the processor to their respective memories, namely memories 62 and 63, which may be part of the main memory locally attached to the respective processor. Although MCs 72 and 82 are shown as integrated into processing elements 70 and 80, in alternative embodiments, the MC logic may be discrete logic outside of processing elements 70 and 80, rather than integrated therein.

[0066] The first processing element 70 and the second processing element 80 can be coupled to the I / O subsystem 90 via PP interconnects 76 and 86, respectively. Figure 11As shown, the I / O subsystem 90 includes PP interfaces 94 and 98. Furthermore, the I / O subsystem 90 includes an interface 92 to couple the I / O subsystem 90 to the high-performance graphics engine 64. In one embodiment, a bus 73 can be used to couple the graphics engine 64 to the I / O subsystem 90. Alternatively, a point-to-point interconnect can couple these components.

[0067] Furthermore, the I / O subsystem 90 may be coupled to the first bus 65 via interface 96. In one embodiment, the first bus 65 may be a Peripheral Component Interconnect (PCI) bus, or a bus such as a Fast PCI bus or another third-generation I / O interconnect bus, although the scope of the embodiment is not limited thereto.

[0068] like Figure 11 As shown, various I / O devices 65a (e.g., biometric scanners, speakers, cameras, sensors) can be coupled to a first bus 66, and a bus bridge 66 can couple the first bus 65a to a second bus 67. In one embodiment, the second bus 67 may be a low pin count (LPC) bus. Various devices can be coupled to the second bus 67, including, for example, a keyboard / mouse 67a, one or more communication devices 67b, and a data storage unit 68 (e.g., a disk drive or other mass storage device), wherein the data storage unit 68 may include code 69 in one embodiment. The illustrated code 69 can implement one or more aspects of the above-described process, including those referenced herein. Figure 1 and Figure 2A-2C The process described in this paper for asymmetric quantization is referenced. Figures 3A-3B and Figure 4 The process described for per-input channel quantization, and references in this paper. Figure 5 and Figures 6A-6B This describes the process for automatic adjustment of blending precision. Code 69 can be implemented as referenced in this article. Figure 7 This describes one or more aspects of the process used to optimize an inference neural network model. The illustrated code 69 can be compared with the previously discussed code 42. Figure 10 Similarly. In addition, audio I / O 67c can be coupled to a second bus 67, and battery 61 can supply power to computing system 60.

[0069] Note that other embodiments are contemplated. For example, replacing Figure 11 The point-to-point architecture allows the system to implement multi-point branch bus or other communication topologies. Alternatively, it can be modified to utilize... Figure 11 The diagram shows more or fewer integrated chips to divide the data. Figure 11 Components.

[0070] Each system and method described above for optimizing inference neural network models, and each of its embodiments (including implementations), can be considered a performance enhancement, at least to the extent that the inference neural network model can include low-precision asymmetric quantization, as described herein, and provides appropriately optimized performance for deployment in any number of possible environments, including those with limited computational and / or memory capabilities. The advantages of the techniques described herein include increased computational efficiency (e.g., more operations per second), reduced memory accesses and lower memory requirements, and improved use of memory caches, all of which result in higher throughput and lower latency.

[0071] Additional notes and examples: Example 1 includes a computing system for optimizing an inference neural network model, comprising a processor and a memory coupled to the processor, the memory including a set of instructions that, when executed by the processor, cause the computing system to generate a quantized neural network, wherein the model weights of the neural network are quantized into signed integer values, and wherein the input layer of the neural network is configured to quantize input values ​​into unsigned integer values, generate a weight accumulation table based on the quantized model weights and the kernel size of the neural network, and generate an output recovery function for the output layer of the neural network based on the weight accumulation table and the kernel size.

[0072] Example 2 includes a computing system as described in Example 1, wherein, in order to generate an output recovery function, the instructions, when executed, cause the computing system to generate a mapping between the output coordinates of the output layer of the neural network and the index of the weight accumulation table.

[0073] Example 3 includes a computing system as described in Example 1, wherein the input layer of the neural network is configured to quantize input values ​​on a per-channel basis, and the model weights of the neural network are quantized on a per-channel basis.

[0074] Example 4 includes a computing system as described in Example 3, wherein the weight accumulation table includes a third dimension, and the value of each level of the third dimension corresponds to each corresponding channel.

[0075] Example 5 includes a computing system as described in Example 1, wherein the neural network includes a plurality of inner layers, and wherein the weight accumulation table is generated based on a plurality of per-layer weight accumulation tables, each per-layer weight accumulation table corresponding to one of the plurality of inner layers of the neural network.

[0076] Example 6 includes a computing system as described in any one of Examples 1 to 5, wherein, when the instructions are executed, the computing system also performs an auto-adjustment process that includes a quantization metric.

[0077] Example 7 includes a semiconductor device for optimizing an inference neural network model, comprising one or more substrates and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic or fixed-function hardware logic, the logic coupled to the one or more substrates generating a quantized neural network, wherein the model weights of the neural network are quantized to signed integer values, and wherein the input layer of the neural network is configured to quantize input values ​​to unsigned integer values, generate a weight accumulation table based on the quantized model weights and the kernel size of the neural network, and generate an output recovery function for the output layer of the neural network based on the weight accumulation table and the kernel size.

[0078] Example 8 includes a semiconductor device as described in Example 7, wherein, in order to generate an output recovery function, the logic coupled to the one or more substrates generates a mapping between the output coordinates of the output layer of the neural network and the index of the weight accumulation table.

[0079] Example 9 includes a semiconductor device as described in Example 7, wherein the input layer of the neural network is configured to quantize input values ​​on a per-channel basis, and the model weights of the neural network are quantized on a per-channel basis.

[0080] Example 10 includes a semiconductor device as described in Example 9, wherein the weighted cumulative table includes a third dimension, and the value of each level of the third dimension corresponds to each corresponding channel.

[0081] Example 11 includes a semiconductor device as described in Example 7, wherein the neural network includes a plurality of inner layers, and wherein the weight accumulation table is generated based on a plurality of per-layer weight accumulation tables, each per-layer weight accumulation table corresponding to one of the plurality of inner layers of the neural network.

[0082] Example 12 includes a semiconductor device as described in any of Examples 7 to 11, wherein the logic coupled to the one or more substrates further performs an auto-adjustment process that includes a quantization measurement.

[0083] Example 13 includes at least one non-transitory computer-readable storage medium comprising a set of instructions for optimizing an inference neural network model, the instructions, when executed by a computing system, causing the computing system to generate a quantized neural network, wherein the model weights of the neural network are quantized into signed integer values, and wherein the input layer of the neural network is configured to quantize input values ​​into unsigned integer values, generate a weight accumulation table based on the quantized model weights and the kernel size of the neural network, and generate an output recovery function for the output layer of the neural network based on the weight accumulation table and the kernel size.

[0084] Example 14 includes at least one non-transitory computer-readable storage medium as described in Example 13, wherein, in order to generate an output recovery function, the instructions, when executed, cause the computing system to generate a mapping between the output coordinates of the output layer of the neural network and the index of the weight accumulation table.

[0085] Example 15 includes at least one non-transitory computer-readable storage medium as described in Example 14, wherein the input layer of the neural network is configured to quantize input values ​​on a per-channel basis, and the model weights of the neural network are quantized on a per-channel basis.

[0086] Example 16 includes at least one non-transitory computer-readable storage medium as described in Example 15, wherein the weight accumulation table includes a third dimension, and the value of each level of the third dimension corresponds to each corresponding channel.

[0087] Example 17 includes at least one non-transitory computer-readable storage medium as described in Example 13, wherein the neural network includes a plurality of inner layers, and wherein the weight accumulation table is generated based on a plurality of per-layer weight accumulation tables, each per-layer weight accumulation table corresponding to one of the plurality of inner layers of the neural network.

[0088] Example 18 includes at least one non-transitory computer-readable storage medium as described in any of Examples 13 to 17, wherein, when the instructions are executed, they also cause the computing system to perform an auto-adjustment process, the auto-adjustment process including quantization metrics.

[0089] Example 19 includes a method of operating a computing device for optimizing an inference neural network model, comprising generating a quantized neural network, wherein the model weights of the neural network are quantized into signed integer values, and wherein the input layer of the neural network is configured to quantize input values ​​into unsigned integer values, generating a weight accumulation table based on the quantized model weights and the kernel size of the neural network, and generating an output recovery function for the output layer of the neural network based on the weight accumulation table and the kernel size.

[0090] Example 20 includes the method as described in Example 19, wherein the output recovery function generates a mapping between the output coordinates of the output layer of the neural network and the index of the weight accumulation table.

[0091] Example 21 includes the method as described in Example 20, wherein the input layer of the neural network is configured to quantize input values ​​on a per-channel basis, and the model weights of the neural network are quantized on a per-channel basis.

[0092] Example 22 includes the method as described in Example 21, wherein the weight accumulation table includes a third dimension, and the value of each level of the third dimension corresponds to each corresponding channel.

[0093] Example 23 includes the method as described in Example 19, wherein the neural network includes a plurality of inner layers, and wherein the weight accumulation table is generated based on a plurality of per-layer weight accumulation tables, each per-layer weight accumulation table corresponding to one of the plurality of inner layers of the neural network.

[0094] Example 24 includes the method described in any of Examples 19 to 23, and further includes performing an automatic adjustment process that includes a quantification measure.

[0095] Example 25 includes an apparatus comprising means for performing the method as described in any one of claims 19-24.

[0096] Therefore, the technique described in this paper improves the performance of inference neural networks through asymmetric quantization, which is achieved by: generating a quantized neural network in which the model weights of the neural network are quantized into signed integer values, and wherein the input layer of the neural network is configured to quantize input values ​​into unsigned integer values; generating a weight accumulation table based on the quantized model weights and the kernel size of the neural network; and generating an output recovery function for the output layer of the neural network based on the weight accumulation table and the kernel size. This technique also performs per-input channel quantization and automatic adjustment of mixed precision. The technique described in this paper is applicable to any number of computing environments, including servers, cloud computing, browsers, and / or any environment with deployed inference neural networks.

[0097] The embodiments are applicable to use with all types of semiconductor integrated circuit (“IC”) chips. Examples of such IC chips include, but are not limited to, processors, controllers, chipset assemblies, programmable logic arrays (PLAs), memory chips, network chips, systems-on-chip (SoCs), SSD / NAND controller ASICs, and so on. Furthermore, in some figures, signal wires are represented by lines. Some may differ to indicate more constituting signal paths, have numerical labels to indicate the number of constituting signal paths, and / or have arrows at one or more ends to indicate the primary direction of information flow. However, this should not be interpreted in a limiting manner. Rather, this added detail can be used in conjunction with one or more exemplary embodiments to facilitate a more readily understood understanding of the circuit. Any represented signal line, whether or not it carries additional information, may substantially comprise one or more signals that may travel in multiple directions and can be implemented using any suitable type of signaling scheme, such as digital or analog lines implemented with differential pairs, fiber optic lines, and / or single-ended lines.

[0098] Example sizes / models / values / ranges may be given, although embodiments are not limited thereto. As manufacturing technologies (e.g., photolithography) mature over time, it is expected that devices with smaller dimensions will be manufactured. Furthermore, for simplicity of illustration and discussion, and to avoid obscuring certain aspects of the embodiments, well-known power / ground connections to the IC chip and other components may or may not be shown in the drawings. Additionally, arrangements may be shown in block diagram form to avoid obscuring the embodiments, also taking into account the fact that the specific details of the implementation of such block diagram arrangements are highly dependent on the platform in which the embodiments are implemented; that is, such specific details should be entirely within the view of those skilled in the art. In the case of setting forth specific details (e.g., circuitry) to describe exemplary embodiments, those skilled in the art will understand that embodiments may be implemented without these specific details, or using variations of these specific details. The specification should therefore be considered illustrative, not restrictive.

[0099] The term "coupling" as used herein may refer to any type of relationship between the components involved, whether direct or indirect, and may apply to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections. Furthermore, unless otherwise indicated, the terms "first," "second," etc., may be used herein merely to facilitate discussion and without any specific temporal or chronological meaning.

[0100] For the purposes of this application and the claims, a list of items joined by the term “one or more of…” may mean any combination of the listed terms. For example, the phrase “one or more of A, B, or C” may mean A, B, C; A and B; A and C; B and C; or A, B, and C.

[0101] Those skilled in the art will understand from the foregoing description that a wide range of techniques can be implemented in various forms. Therefore, although embodiments have been described with reference to specific examples, the true scope of the embodiments should not be limited thereto, as other modifications will become apparent to those skilled in the art upon studying the drawings, specification, and appended claims.

Claims

1. At least one computer-readable storage medium, comprising a set of instructions for managing a runtime computing environment, the instructions causing the computing system to perform operations when executed by the computing system, the operations including: Generate a quantized neural network, wherein symmetric quantization is used to quantize the model weights of the neural network into signed integer values, and wherein the input layer of the neural network is configured to quantize the input values ​​into unsigned integer values ​​using asymmetric quantization with bias or offset. The compensation term is determined based on the sum of the quantized model weights and the bias or offset; as well as The output of the input layer of the quantized neural network is generated by applying the compensation term to the cumulative product of the quantized input value and the quantized model weights.

2. The computer-readable storage medium of claim 1, wherein the model weights are pre-computed once to generate the output for the input layer.

3. The computer-readable storage medium of claim 1, wherein the quantized neural network is configured as an inference model for performing inference operations based on the quantized model weights and the quantized input values.

4. The computer-readable storage medium of claim 1, wherein the bias or offset includes a zero-point offset.

5. The computer-readable storage medium of claim 1, wherein the quantized model weights are stored in a data structure in the memory of the computing system.

6. The computer-readable storage medium of claim 5, wherein the data structure includes a weighted cumulative table.

7. The computer-readable storage medium of claim 6, wherein the weight accumulation table comprises an accumulation of integer weights within a range of kernel indexes.

8. A calculation method, comprising: Generate a quantized neural network, wherein symmetric quantization is used to quantize the model weights of the neural network into signed integer values, and wherein the input layer of the neural network is configured to quantize the input values ​​into unsigned integer values ​​using asymmetric quantization with bias or offset. The compensation term is determined based on the sum of the quantized model weights and the bias or offset; as well as The output of the input layer of the quantized neural network is generated by applying the compensation term to the cumulative product of the quantized input value and the quantized model weights.

9. The calculation method of claim 8 further includes pre-computing the model weights once to generate the output for the input layer.

10. The computation method of claim 8, further comprising configuring the quantized neural network as an inference model, the inference model being used to perform inference operations based on the quantized model weights and the quantized input values.

11. The calculation method of claim 8, wherein the bias or offset includes a zero-point offset.

12. The calculation method of claim 8 further includes storing the quantized model weights in a data structure in the memory of the computing system.

13. The calculation method of claim 12, wherein the data structure includes a weighted cumulative table.

14. The calculation method of claim 13, wherein the weight accumulation table comprises the accumulation of integer weights within the range of the kernel index.

15. A computing system, comprising: Memory, which stores instructions; as well as One or more processors, The instructions thereon are executed by the one or more processors to cause the one or more processors to perform the computation method as described in any one of claims 8-14.

16. A computing device comprising means for performing the computing method as described in any one of claims 8-14.

17. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the computation method as described in any one of claims 8-14.

Images