Quantized neural network training and inference

Abstract

The present disclosure relates to quantization of neural network training and inference. One embodiment of a computer-implemented method for processing a neural network includes receiving a first quantized matrix corresponding to a portion of a multi-dimensional input tensor and having been quantized based on a first scale factor; and performing one or more computational operations using the first quantized matrix and the first scale factor to generate one or more data values corresponding to a first portion of a multi-dimensional output tensor.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 238,036, filed August 27, 2021, entitled “TECHNIQUES FOR QUANTIZED FIXED-POINT TRAINING WITH FINE-GRAINED SCALING AND ADAPTIVE PRECISION”. The subject matter of that related application is incorporated herein by reference. Technical Field

[0003] The embodiments of this disclosure generally relate to computer science and artificial intelligence, and more specifically, to techniques for quantifying neural network training and inference. Background Technology

[0004] Deep neural networks typically consist of many computationally intensive layers, such as convolutional layers and / or fully connected layers. Deep neural networks also often use full-precision algorithms (e.g., using 32-bit floating-point data) for training and deployment. Therefore, executing a deep neural network can involve significant computational complexity, latency, and memory consumption. To reduce the processing time and memory footprint associated with deep neural networks, the weights and / or activations of the network can be quantized. For example, 32-bit floating-point data values ​​used for weights and / or activations can be mapped to 16-bit floating-point data values ​​or 8-bit integer data values, which reduces the bit width of the data values. In turn, reducing the bit width of the data values ​​can reduce the complexity of the various computations performed during the training and deployment of the deep neural network (e.g., matrix multiplication operations), as well as the amount of memory required to store the data values.

[0005] Typically, to quantize the weights and activations of a neural network, a scaling factor is applied to both the weight tensor representing the weights and the activation tensor representing the activations. However, since the same scaling factor is shared across many dimensions of a given tensor, the precision of the individual elements in the scaling tensor decreases, thereby increasing the amount of error introduced by quantization, which in turn reduces the overall accuracy of the deep neural network.

[0006] To improve the accuracy of deep neural networks, different scaling factors can be applied to different vectors along a single dimension of the weight tensor and / or activation tensor. This allows for the use of more appropriate scaling factors for different vectors, reducing the amount of error introduced by quantization and, in turn, improving the overall accuracy of the deep neural network. However, unlike neural network inference operations, which typically involve computations using only matrices, neural network training operations typically involve computations using both matrices and their transposes. It is noteworthy that when a matrix is ​​transposed, each column of the transpose corresponds to multiple different scaling factors. Since each of these multiple different scaling factors must be read individually from memory, computations involving the transpose increase the overall processing time. For example, if reading a scaling factor from memory takes one clock cycle, then for a 4x4 transpose matrix, reading the relevant scaling factor from memory for each column of the transpose matrix will take four clock cycles. Therefore, applying different scaling factors along a single dimension of the weight tensor and / or activation tensor can significantly increase processing time when training a deep neural network.

[0007] As mentioned above, what is needed in the art is a more efficient technique for quantizing weights and activations for neural networks. Summary of the Invention

[0008] One embodiment of this disclosure illustrates a computer implementation method for processing a neural network. The method includes receiving a first quantization matrix corresponding to a portion of a multidimensional input tensor and having been quantized based on a first scaling factor. The method further includes performing one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to a first portion of the multidimensional output tensor.

[0009] At least one technical advantage of the disclosed technique over the prior art is that neural network training operations can be performed more efficiently using the disclosed technique. In this regard, using the disclosed technique, different scaling factors are applied to different submatrices along multiple dimensions of a tensor, rather than different vectors along a single dimension of the tensor. Therefore, the same scaling factor can be applied to submatrices within a matrix and to corresponding submatrices within the corresponding transpose matrix. Thus, compared to prior art methods, performing computations using the transpose matrix requires fewer read operations, thereby reducing processing time when training neural networks. These technical advantages provide one or more technical improvements over prior art methods. Attached Figure Description

[0010] To gain a more detailed understanding of the features of the various embodiments described above, the briefly summarized inventive concept can be described in more specific terms with reference to various embodiments (some of which are illustrated in the accompanying drawings). However, it should be noted that the accompanying drawings illustrate only typical embodiments of the inventive concept and are therefore not intended to limit the scope in any way; other equally effective embodiments exist.

[0011] Figure 1 This is a block diagram illustrating a computer system configured to implement one or more aspects of various embodiments;

[0012] Figure 2 Included according to various embodiments Figure 1 A block diagram of the parallel processing unit in the parallel processing subsystem;

[0013] Figure 3A Example inputs and generated outputs are shown according to various embodiments when performing one or more forward propagation operations;

[0014] Figure 3B Example inputs and generated outputs are shown according to various embodiments when performing one or more backpropagation operations;

[0015] Figure 4 This is a flowchart of method steps for retrieving data from a group buffer and storing data into a group buffer according to various embodiments; and

[0016] Figure 5 This is a flowchart of method steps for performing one or more neural network computation operations using quantized data, according to various embodiments. Detailed Implementation

[0017] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the various embodiments. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0018] System Overview

[0019] Figure 1 A block diagram of a computer system 100 configured to implement one or more aspects of various embodiments is shown. As illustrated, the computer system 100 includes, but is not limited to, a central processing unit (CPU) 102 and a system memory 104 coupled to a parallel processing subsystem 112 via a memory bridge 105 and a communication path 113. The memory bridge 105 is further coupled to an I / O (input / output) bridge 107 via a communication path 106, and the I / O bridge 107 is in turn coupled to a switch 116.

[0020] In operation, I / O bridge 107 is configured to receive user input from input device 108 (such as a keyboard or mouse) and forward the input to CPU 102 for processing via communication path 106 and memory bridge 105. Switch 116 is configured to provide connectivity between I / O bridge 107 and other components of computer system 100, such as network adapter 118 and various add-on cards 120 and 121.

[0021] As also shown in the figure, I / O bridge 107 is coupled to system disk 114, which can be configured to store content, applications, and data for use by CPU 102 and parallel processing subsystem 112. Generally, system disk 114 provides non-transitory storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROMs (optical disc read-only memory), DVD-ROMs (digital versatile discs), Blu-ray, HD-DVDs (high-definition DVDs), or other magnetic, optical, or solid-state storage devices. Finally, although not explicitly shown, other components (such as universal serial bus or other port connections, optical disc drives, digital versatile disc drives, film recording devices, etc.) may also be connected to I / O bridge 107.

[0022] In various embodiments, memory bridge 105 may be a northbridge chip, and I / O bridge 107 may be a southbridge chip. Furthermore, communication paths 106 and 113, as well as other communication paths, can be implemented within computer system 100 using any technically suitable protocol (including but not limited to AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art).

[0023] In some embodiments, the parallel processing subsystem 112 includes a graphics subsystem that supplies pixels to a display device 110, which can be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, etc. In this embodiment, the parallel processing subsystem 112 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. Such circuitry may be combined across one or more parallel processing units (PPUs) included in the parallel processing subsystem 112, which is configured to perform graphics and video processing operations. In other embodiments, the parallel processing subsystem 112 includes a deep learning subsystem for training neural network models and / or utilizing trained neural network models. In such embodiments, the parallel processing subsystem 112 incorporates circuitry optimized for performing neural network training and neural network inference operations, for example, including a multiplier-accumulator (MAC unit). Figure 2As discussed in further detail, such circuitry can be combined across one or more PPUs included in the parallel processing subsystem 112, which are configured to perform neural network training and / or inference operations. In yet another embodiment, the parallel processing subsystem 112 incorporates circuitry optimized for general and / or computational processing. Again, such circuitry can be combined across one or more PPUs included in the parallel processing subsystem 112, which are configured to perform such general and / or computational operations.

[0024] In various embodiments, the one or more PPUs included in the parallel processing subsystem 112 are configured to perform graphics processing, neural network training and inference, general processing, and computational processing operations. System memory 104 includes at least one device driver 103 configured to manage the processing operations of one or more PPUs in the parallel processing subsystem 112. System memory 104 also includes a software application 125 that executes on CPU 102 and can issue commands to control the operation of the PPUs.

[0025] In various embodiments, the parallel processing subsystem 112 may be connected with Figure 1 One or more other elements can be integrated to form a single system. For example, the parallel processing subsystem 112 can be integrated with the CPU 102 and other interconnect circuitry on a single chip to form a system-on-a-chip (SoC).

[0026] It should be understood that the system shown herein is illustrative, and variations and modifications are possible. The connection topology (including the number and arrangement of bridges, the number of CPUs 102, and the number of parallel processing subsystems 112) can be modified as needed. For example, in some embodiments, system memory 104 may be directly connected to CPU 102 instead of being connected to CPU 102 via memory bridge 105, and other devices will communicate with system memory 104 via memory bridge 105 and CPU 102. In other alternative topologies, parallel processing subsystems 112 may be connected to I / O bridge 107 or directly to CPU 102 instead of being connected to memory bridge 105. In other embodiments, I / O bridge 107 and memory bridge 105 may be integrated into a single chip rather than existing as one or more discrete devices. Finally, in some embodiments, Figure 1 One or more of the components shown may be absent. For example, it is possible to eliminate the direct connection of switch 116, network adapter 118, and add-on cards 120, 121 to I / O bridge 107.

[0027] Figure 2 According to various embodiments Figure 1A block diagram of a parallel processing unit (PPU) 202 included in the parallel processing subsystem 112 is shown. As shown, the PPU 202 communicates with a group buffer 204. The PPU 202 and the group buffer 204 may be implemented using one or more integrated circuit devices, such as programmable processors, application-specific integrated circuits, memory devices, or any other technically feasible method. The PPU 202 may communicate with the group buffer 204 in any technically feasible manner. In some embodiments, the PPU 202 communicates with the group buffer 204 via a data router (not shown).

[0028] In some embodiments, PPU 202 includes a neural network processing element (PE) configured to perform various operations related to the training and inference of a neural network based on input data provided by CPU 102 and / or system memory 104. In this case, group buffer 204 stores neural network parameters used to perform the neural network training and inference operations, such as the input activations, output activations, weights, input gradients, and / or output gradients of each layer of the neural network. As shown, group buffer 204 includes a unified buffer 210 storing activations 212, weights 214, and gradients 216 associated with the neural network.

[0029] In operation, group buffer 210 retrieves data (e.g., activations 212, weights 214, and / or gradients 216) stored in unified buffer 210 and transmits this data as input data 206 to PPU 202. PPU 202 receives input data 206 and performs one or more computational operations on the input data 206 for neural network training and / or inference to generate corresponding output data 208. As discussed in further detail below, the one or more computational operations include, for example, matrix multiplication, vector dot product, data scaling, and data quantization. After generating the output data 208, PPU 202 transmits the output data 208 to group buffer 204. Group buffer 204 stores the output data in unified buffer 210. The stored output data 208 can then be retrieved and transmitted to PPU 202 as additional input data 206.

[0030] In some embodiments, group buffer 204 identifies portions of data stored in unified buffer 210 that are to be transmitted to PPU 202 for processing. Identifying the portions of data to be transmitted to PPU 202 may include, for example, identifying input data for the current layer of a neural network, identifying input data for the current neural network operation, etc.

[0031] In some embodiments, the unified buffer 210 includes multiple sets of data, each associated with a different layer of the neural network. The group buffer 204 identifies one or more sets of data corresponding to the current layer of the neural network. For example, activations 212 may include different sets of output activations, each generated by computational operations performed for different layers of the neural network. The group buffer 204 identifies one or more sets of output activations generated by one or more previous layers based on the current layer of the neural network, which should be used as input activations for the current layer of the neural network. The group buffer 204 retrieves the one or more sets of output activations from the unified buffer 210. As another example, weights 214 may include different sets of weights, each associated with a different layer of the neural network. The group buffer 204 identifies the sets of weights associated with the current layer of the neural network based on the current layer. The group buffer 204 retrieves the identified sets of weights from the unified buffer 210.

[0032] In some embodiments, group buffer 204 identifies one or more datasets used as inputs to the neural network computation of the current layer based on the type of neural network processing performed by PPU 202. For example, neural network inference includes forward propagation, while neural network training includes forward propagation and backward propagation. Forward propagation includes performing computation between an input activation set and a weight set to generate an output activation set. Backward propagation includes performing computation between an output activation gradient set and a transposed weight set to generate an input activation gradient set, and performing computation between an output activation gradient set and a transposed input activation set to generate a weight gradient set.

[0033] Figure 3A The inputs and generated outputs used during the execution of forward propagation 300 are shown according to various embodiments. Figure 3A As shown, forward propagation 300 involves performing computations between input activations 302 and weights 304 to generate output activations 306. The input activations 302, weights 304, and output activations 306 correspond to specific layers of the neural network. That is, each layer of the neural network may involve different sets of input activations 302 and weights 304, and generate a corresponding set of output activations 306. When forward propagation 300 is performed on a specific layer of the neural network, a group buffer 204 retrieves a portion of activations 212 corresponding to the input activations 302 of the specific layer. Furthermore, the group buffer 204 retrieves a portion of weights 214 corresponding to the weights 304 of the specific layer. The group buffer 204 transmits the retrieved portions of activations 212 and weights 214 as inputs 206 to the PPU 202.

[0034] Figure 3B The inputs and generated outputs used during the execution of backpropagation 310 are shown according to various embodiments. Figure 3B As shown, backpropagation 310 generates two output sets: input activation gradient 316 and weight gradient 320. A computation is performed between the output activation gradient 312 and the transposed weight 304 to generate the input activation gradient 316. Further computation is performed between the output activation gradient 312 and the transposed input activation 302 to generate the weight gradient 320. The output activation gradient 312, weight 304, input 302, input activation gradient 316, and weight gradient 320 correspond to specific layers of the neural network. That is, each layer of the neural network can involve different sets of output activation gradient 312, transposed weight 304, and transposed input 302, and generate a corresponding set of input activation gradient 316 and weight gradient 320.

[0035] When backpropagation 310 is performed on a specific layer of the neural network, group buffer 204 determines whether the current operation is to generate input activation gradient 316 or weight gradient 320. If the current operation is to generate input activation gradient 316, group buffer 204 retrieves a portion of gradient 216 corresponding to the output activation gradient 312. Additionally, group buffer 204 retrieves a portion of weight 214 corresponding to the weight 304. In some embodiments, group buffer 204 transposes the retrieved portion of weight 214. If the current operation is to generate weight gradient 320, group buffer 204 retrieves the portion of gradient 216 corresponding to the output activation gradient 312 and a portion of activation 212 corresponding to the input activation 302. In some embodiments, group buffer 204 transposes the retrieved portion of activation 212.

[0036] like Figure 2As shown, group buffer 204 also includes data path unit 218. Data path unit 218 is configured to receive data and perform one or more data processing operations on the received data. The one or more data processing operations include, for example, element-wise subtraction, truncation, and transposition. In some embodiments, data path unit 218 receives data retrieved from unified buffer 210 and performs one or more data processing operations on the data before group buffer 204 transmits the data to PPU 202. For example, as described above, backpropagation includes performing computation using transposed weights and transposed input activations. Group buffer 204 retrieves a portion of weights 214 and sends the weights to data path unit 218. Data path unit 218 transposes the weights to generate transposed weights, and group buffer 204 transmits the transposed weights to PPU 202. Data path unit 218 is configured to transpose data using any suitable techniques and / or algorithms. In some embodiments, data path unit 218 transposes data by transmitting data to PPU 202 in a transposed order.

[0037] In some embodiments, data path unit 218 receives output data 208 transmitted by PPU 202 and performs one or more data processing operations on the output data 208. The data processing operations may include, for example, updating data stored in unified buffer 210 based on the output data 208, processing the output data 208 before storing it in unified buffer 210, etc. For example, during backpropagation 310, PPU 202 generates a weight gradient 320. Group buffer 204 receives the output 208 including the weight gradient 320. Group buffer 204 transmits the output 208 to data path unit 218. Data path unit 218 identifies a set of weights included in a weight 214 corresponding to the weight gradient 320 and performs element-wise subtraction between the weight set and the weight gradient 320 to update the weight set.

[0038] In various embodiments, activation 212, weights 214, and / or gradient 216 include quantized data. As discussed in further detail below, one or more data values, such as those included in activation 212, weights 214, and / or gradient 216, are quantized from a high-precision, large-bit-width data type to a smaller-bit-width integer. The unified buffer 210 also includes a scaling factor for quantizing the one or more data values. In various embodiments, if the data retrieved from the unified buffer 210 includes quantized data, the group buffer 204 also retrieves a scaling factor for generating the quantized data.

[0039] Fine-grained scaling for neural network quantization

[0040] Quantization scales a high-precision value within a given range to a low-precision value within a different range. For example, it scales a high-precision number x to a low-precision number x. q Example equations are given by equation (1):

[0041]

[0042] In equation (1), s represents the scaling factor, and Q(a,b) represents the function that quantizes the value a into a b-bit integer. Therefore, in equation (1), the high-precision number x is scaled by s, and the scaled value is quantized into an N-bit integer x. q The data values ​​included in activation 212, weight 214, and / or gradient 216 can be quantized, for example, using equation (1) or other suitable quantization techniques. Different scaling factors can be used to quantize each tensor included in activation 212, weight 214, and gradient 216. Additionally, different scaling factors can be used to quantize different parts of each tensor. The scaling factors can be any suitable numerical data format, such as integers, floating-point numbers, etc.

[0043] Each layer of a neural network comprises activations, weights, and gradients (if neural network training is performed), all of which are multidimensional tensors (e.g., input activation 302, weights 304, input activation 302, and output activation gradient 312). To quantize the tensor, it is subdivided into multiple matrices (also called submatrices) along its multiple dimensions. For each matrix, a scaling factor is determined and used to quantize each element contained within that matrix. Therefore, different scaling factors can be used to quantize different portions of the tensor. Furthermore, because a single scaling factor is used to quantize the elements of a given matrix, when the given matrix (or a larger matrix containing the given matrix) is transposed, the given matrix and the transposed matrix share the same single scaling factor. Therefore, reading the scaling factor from the matrix and the transposed matrix from memory takes the same amount of time and uses the same number of read operations.

[0044] In some embodiments, the tensor is subdivided along the tensor's dimensions based on the tensor's type (e.g., activation tensor, weight tensor, activation gradient tensor, or weight gradient tensor). For example, refer to... Figure 3A Input activation 302 includes dimensions B, H, W, and C, and weights 304 include dimensions K, R, S, and C. Input activation 302 can be subdivided along the dimensions B and C, while weights 304 can be subdivided along the dimensions K and C. As another example, refer to... Figure 3BThe output activation gradient 312 includes dimensions B, P, Q, and K; the weights 304 include dimensions K, R, S, and C; and the input activation 302 includes dimensions B, H, W, and C. The output activation gradient 312 can be subdivided along the dimensions B and K, the weights 304 can be subdivided along the dimensions K and C, and the input activation 302 can be subdivided along the dimensions B and C.

[0045] In some embodiments, each tensor is subdivided into multiple matrices of the same size. Furthermore, in some embodiments, each matrix has the same number of rows and columns. For example, the input activation 302 and weights 304 can each be subdivided into multiple matrices with V columns and V rows (V... 2 (Matrix). In some embodiments, the number of rows and columns included in each matrix (i.e., the size of V) corresponds to the size of the vector multiplication-accumulation (MAC) unit 226 included in the PPU 202. As discussed in further detail below, each vector MAC unit 226 is configured to perform computation between two sets of input elements of a given size. The size of the matrix into which each tensor is subdivided is chosen to match the number of input pairs received as input by the vector MAC unit 226. As a result, the vector MAC unit 226 can perform computation on an entire row or an entire column of the matrix.

[0046] In some embodiments, each quantization matrix, along with its corresponding scaling factor, is stored in a unified buffer 210. When a group buffer 204 retrieves a quantization matrix from the unified buffer 210, it also retrieves the corresponding scaling factor. Therefore, the quantization matrix and its corresponding scaling factor included in the input 206 are transmitted to the PPU 202. As explained in further detail below, the PPU 202 uses the quantized values ​​to perform neural network training and / or inference computations.

[0047] Quantizing Neural Network Inference and Training

[0048] PPU 202 receives input data 206 sent by group buffer 204 and performs one or more computational operations on the input data 206 for neural network training and / or neural network inference. In some embodiments, PPU 202 receives the input data 206 via a data router or other suitable communication path within parallel processing subsystem 112. Figure 2 As shown, PPU 202 includes one or more data buffers, such as data buffer 220(1) and data buffer 220(2), one or more vector multiplication-accumulation (MAC) units 226, accumulation unit 228 and post-processing unit 230.

[0049] In some embodiments, the input data 206 received from group buffer 204 includes a first input dataset corresponding to a first type of input data and a second input dataset corresponding to a second type of input data. The first type of input data and the second type of input data can vary depending on the specific neural network training or inference computation being performed and the type of the target output. For example, for forward propagation, a computation is performed between input activations and weights to generate output activations. PPU 202 can be configured to store weights in data buffer 220(1) and input activations in data buffer 220(2). As another example, backpropagation includes a computation between output activation gradients and transposed input activations to generate weight gradients. PPU 202 can be configured to store transposed input activations in data buffer 220(1) and output activation gradients in data buffer 220(2).

[0050] In some embodiments, the input data 206 received from group buffer 204 includes only the first input dataset corresponding to the first type of input data. PPU 202 receives second input data 206 from group buffer 204, the second input data 206 including the second input dataset corresponding to the second type of input data. In such embodiments, the number of input datasets corresponding to the first type of input data and the number of input datasets corresponding to the second type of input data may be different. For example, PPU 202 may receive a single weight set and multiple input activation sets. PPU 202 uses each input activation set and the same weight set to perform computations. As another example, PPU 202 may receive an output gradient set and a transposed weight set. PPU 202 performs computations on the output activation gradient set and the transposed weight set to generate an input activation gradient set. PPU 202 then receives the transposed input activation set and performs computations on the same output activation gradient set and the transposed input activation set to generate a weight gradient set.

[0051] PPU 202 stores input data 206 in data buffer 220(1) and / or data buffer 220(2). In some embodiments, data buffers 220(1) and 220(2) each store different types of input data. PPU 202 determines where to store the input data 206 based on the type of the input data. For example, PPU 202 may be configured to store weights in data buffer 220(1) and input activations in data buffer 220(2). If the input data 206 includes weights, PPU 202 stores the weights in data buffer 220(1). If the input data 206 includes input activations, PPU 202 stores the input activations in data buffer 220(2). In some embodiments, input data 206 includes multiple types of input data. PPU 202 divides the input data 206 into multiple portions, each portion corresponding to a different type of input data. For each part of the input data 206, PPU 202 determines whether to store the part of the input data 206 in data buffer 220(1) or data buffer 220(2).

[0052] In various embodiments, when performing forward propagation, PPU 202 stores weights in data buffer 220(1) and input activations in data buffer 220(2). When performing backward propagation, PPU 202 stores transposed input activations or transposed weights in data buffer 220(1) and output activation gradients in data buffer 220(2), depending on whether PPU 202 is calculating input activation gradients or weight gradients.

[0053] If the input data 206 includes quantized data, then the input data 206 also includes one or more scaling factors for generating the quantized data. The PPU 202 stores the quantized data and the corresponding scaling factors together in data buffers 220(1) and / or 220(2). Figure 2 As shown, data buffer 220(1) stores one or more data vectors 222(1) and one or more corresponding scaling factors 224(1), and data buffer 220(2) stores one or more data vectors 222(2) and one or more corresponding scaling factors 224(2). In some embodiments, the input data stored in data buffer 220(1) and the input data stored in data buffer 220(2) both include quantized data. In other embodiments, one of the input data stored in data buffer 220(1) or data buffer 220(2) includes quantized data, while the input data stored in the other data buffer 220 does not include quantized data.

[0054] As described above, to quantize the tensor, the tensor is divided into multiple matrices, and each matrix is ​​quantized using a corresponding scaling factor. In some embodiments, each matrix and its corresponding scaling factor are stored in the same row of a buffer (e.g., a uniform buffer 210, a data buffer 220(1), and a data buffer 220(2)). For example, each data vector 222(1) corresponds to a different matrix, and each scaling factor 224(1) corresponds to a scaling factor used to quantize the corresponding matrix. Each data vector 221(1) and its corresponding scaling factor 224(1) are stored in the same row of the data buffer 220(1). When reading data from the buffer, the elements of the matrix can be read together with the corresponding scaling factors.

[0055] In some embodiments, data buffer 220(1) includes multiple memory banks. One or more memory banks store matrix elements, and another memory bank stores scaling factors. In various embodiments, each memory bank storing matrix elements stores different rows of the matrix. For example, the first row of each matrix is ​​stored in a first memory bank, the second row of each matrix is ​​stored in a second memory bank, and so on. In other embodiments, each memory bank storing matrix elements stores different columns of the matrix. For example, the first column of each matrix is ​​stored in a first memory bank, the second column of each matrix is ​​stored in a second memory bank, and so on.

[0056] To perform neural network computations, one or more vector MAC units 226 read input data from data buffers 220(1) and 220(2) and perform one or more computations on the input data. In some embodiments, each vector MAC unit 226 is configured to read a first set of input data from data buffer 220(1) and a second set of input data from data buffer 220(2). For example, the vector MAC unit 226 reads a data vector 222(1) and a scaling factor 224(1) from data buffer 220(1), and a data vector 222(2) and a scaling factor 224(2) from data buffer 220(2). The vector MAC unit 226 uses elements in data vector 222(1), elements in data vector 222(2), scaling factor 224(1), and / or scaling factor 224(2) to perform one or more computations.

[0057] In some embodiments, vector MAC unit 226 multiplies each element in data vector 222(1) with the corresponding element in data vector 222(2). Vector MAC unit 226 sums the product values ​​to produce a partial sum. That is, vector MAC unit 226 calculates the vector dot product between data vector 222(1) and data vector 222(2). Furthermore, vector MAC unit 226 multiplies scaling factors 224(1) and 224(2) to generate a scaling factor product. Vector MAC unit 226 scales the partial sum based on scaling factors 224(1) and 224(2) by multiplying the partial sum with the scaling factor product to generate a scaled partial sum. In some embodiments, vector MAC unit 226 rounds the scaling factor product before multiplying the scaling factor product with the partial sum.

[0058] In some embodiments, PPU 202 includes a plurality of vector MAC units 226. Each vector MAC unit 226 reads a different input dataset from data buffer 220(1) and / or data buffer 220(2). Therefore, the vector MAC units 226 can perform computations on different input dataset pairs in parallel or substantially in parallel. The vector MAC units 226 may each read a different input dataset from data buffer 220(1), read the same input dataset from data buffer 220(2), read the same input dataset from data buffer 220(1), read a different input dataset from data buffer 220(2), or read different input datasets from data buffer 220(1) and data buffer 220(2).

[0059] In some embodiments, data buffer 220(1) and / or data buffer 220(2) include multiple memory groups, including one or more memory groups storing elements of the quantization matrix. Each vector MAC unit 226 reads a set of matrix elements from the same row of data buffer 220(1) and / or data buffer 220(2), but from different memory groups. In some embodiments, each of the one or more memory groups stores different rows of the quantization matrix. When each vector MAC unit 226 reads from a different memory group, each vector MAC unit 226 reads a different row of the quantization matrix from that data buffer. In some embodiments, each of the one or more memory groups includes different columns of the quantization matrix. When each vector MAC unit 226 reads from a different memory group, each vector MAC unit 226 reads a different column of the quantization matrix from that data buffer. Additionally, each vector MAC unit 226 may read a scaling factor from the same memory group. Because the same scaling factor is used to quantize the elements of the matrix, different rows and different columns of the quantization matrix correspond to the same scaling factor.

[0060] As described above, the number of rows and columns included in the quantization matrix can match the number of elements that each vector MAC unit 226 is configured to read from data buffer 220(1) or data buffer 220(2). For example, if the vector MAC unit 226 is configured to perform V multiplication operations in parallel, then each quantization matrix also has V rows and V columns. Furthermore, in some embodiments, the PPU 202 includes a number of vector MAC units 226 of the same size as each vector MAC unit 226. If each vector MAC unit 226 reads different rows of the matrix, or if each vector MAC unit 226 reads different columns of the matrix, then the PPU 202 can perform computations on the entire matrix using the vector MAC units 226. Referring to the example above, the PPU 202 includes V vector MAC units 226. The V elements of the matrix are read by a total of V vector MAC units 226, corresponding to V 2 The number of elements is also the size of the matrix.

[0061] Accumulation unit 228 receives the outputs generated by one or more vector MAC units 226, such as scaled partial sums, and accumulates the outputs. In some embodiments, the output generated by each vector MAC unit 226 corresponds to a specific position or element of the output tensor. Accumulation unit 228 accumulates the output corresponding to each position or element of the output tensor. After all input elements have been processed by the one or more vector MAC units 226, the accumulated outputs form the complete output tensor. For example, refer to... Figure 3AEach scaled partial sum may correspond to a different element of the output activation 306. The accumulator 228 accumulates the scaled partial sums generated by the one or more vector MAC units 226 corresponding to each element of the output activation 306. The output activation 306 is then completed after each element of the input activation 302 and weights 304 has been processed by the one or more vector MAC units 226. In some embodiments, the accumulator includes an output address generator that calculates the storage address of each partial sum. The accumulator uses the storage address determined by the output address generator to determine which part of the output tensor each partial sum belongs to.

[0062] Post-processing unit 230 receives the accumulated output from accumulating unit 228, such as an output activation tensor, a weight gradient tensor, or an input activation gradient tensor. In some embodiments, post-processing unit 230 performs one or more post-processing operations on the accumulated output before PPU 202 transmits the accumulated output to group buffer 204. PPU 202 transmits the processed output as output 208 to group buffer 204.

[0063] In some embodiments, the one or more post-processing operations include quantizing each element included in the accumulated output. As described above, a tensor can be partitioned into multiple matrices, each element of which is quantized to an N-bit integer based on a scaling factor. Post-processing unit 230 partitions the output tensor into multiple matrices and quantizes each matrix using a corresponding scaling factor. In some embodiments, post-processing unit 230 calculates the scaling factor of each matrix based on the elements included in the matrix. Any suitable algorithm or equation can be used to calculate the scaling factor of a matrix. Equation (2) gives an example function for calculating the scaling factor of a matrix:

[0064]

[0065] In equation (2), s, i, X(i), and N correspond to the scaling factor of the matrix, the number of elements in the matrix, the i-th element of the matrix, and the bit width to which the value is quantized, respectively. Therefore, in equation (2), the scaling factor of the matrix is ​​calculated based on the target bit width and the maximum value included in the matrix.

[0066] Post-processing unit 230 quantizes the elements included in each matrix based on the calculated scaling factor. For example, the quantization of elements can be performed using equation (1) discussed above. In some embodiments, during neural network training, the bit width N to which the value is quantized varies depending on the current epoch. A first bit width can be used for multiple epochs of neural network training, and a second bit width can be used after a threshold number of epochs has been reached. For example, the value can be quantized to a 4-bit integer from epoch 0 to epoch 75, and the value can be quantized to an 8-bit integer from epoch 76 to 86.

[0067] In some embodiments, post-processing unit 230 scales each element of the output tensor based on a per-layer scaling factor before quantizing the elements of the output tensor. Each layer of the neural network can be associated with a per-layer scaling factor, in addition to the scaling factor associated with different parts of the tensor. Post-processing unit 230 uses the per-layer scaling factor to scale each element of the output tensor. The per-layer scaling factor is the scaling factor associated with the current layer of the neural network, i.e., the scaling factor associated with the output tensor. In some embodiments, post-processing unit 230 receives the per-layer scaling factor for scaling the output tensor, for example, as part of the input data 206. In some embodiments, post-processing unit 230 computes the per-layer scaling factor based on the values ​​contained in the output tensor. Any suitable algorithm or equation can be used to compute the per-layer scaling factor. Equation (3) gives an example function for computed to the per-layer scaling factor:

[0068]

[0069] As shown in equation (3), the scaling factor s of each layer is calculated based on the element k of the neural network layer L and the bit width N used for quantization. layer .

[0070] In some embodiments, one or more post-processing operations include processing the elements of the output tensor using a nonlinear function. The post-processing unit 230 may process the elements of the output tensor using the nonlinear function before quantizing them. In some embodiments, the post-processing unit 230 scales the elements of the output tensor using the per-layer scaling factor and processes the scaled elements using the nonlinear function.

[0071] PPU 202 transmits the processed output tensor as output 208 to group buffer 204. In some embodiments, PPU 202 also includes one or more scaling factors in output 208. The one or more scaling factors include, for example, scaling factors for quantizing different portions of the output tensor and / or scaling factors for each layer of the output tensor.

[0072] Figure 4 It is according to various embodiments for use from Figure 2 Group buffer 204 retrieves data and stores the data. Figure 2 A flowchart of the method steps for group buffer 204. Although referenced Figure 1 and Figure 2 The system described herein describes the method steps, but those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of this invention.

[0073] like Figure 4As shown, method 400 begins at step 402, where group buffer 204 identifies a set of input data 206 to be sent to PPU 202 and retrieves the set of input data 206 from unified buffer 210. Identification of the set of input data 206 is performed in a manner similar to that discussed above with respect to group buffer 204. Identifying the input dataset includes, for example, identifying input data for the current layer of a neural network, input data for the current neural network operation, etc. Group buffer 204 determines one or more memory addresses in unified buffer 210 corresponding to the input data and retrieves the input data from unified buffer 210 based on the one or more memory addresses.

[0074] In some embodiments, the unified buffer 210 includes multiple datasets, each associated with a different layer of the neural network. The group buffer 204 identifies one or more datasets corresponding to the current layer of the neural network.

[0075] In some embodiments, group buffer 204 identifies one or more datasets used as inputs to the neural network computation of the current layer based on the type of neural network computation performed by PPU 202. If PPU 202 is performing forward propagation computation, group buffer 204 identifies weight set 214 and activation set 212. If PPU 202 is performing backpropagation computation to generate input activation gradients, group buffer 204 identifies weight set 214 and activation gradient set 216. If PPU 202 is performing backpropagation computation to generate weight gradients, group buffer 204 identifies activation set 212 and activation gradient set 216.

[0076] In step 404, group buffer 204 determines whether to transpose the input data 206. The determination of whether to transpose the input data 206 is based on the type of neural network computation performed by PPU 202. If PPU 202 is performing a forward propagation computation, the computation is performed using untransposed weights and activations. Therefore, if PPU 202 is performing a forward propagation computation, group buffer 204 determines that the input data 206 should not be transposed.

[0077] If PPU 202 is performing backpropagation computation to generate input activation gradients, the computation is performed using transposed weights. Therefore, if PPU 202 is performing backpropagation computation to generate input activation gradients and input data 206 includes weights, group buffer 204 determines that the weights included in input data 206 should be transposed. If PPU 202 is performing backpropagation computation to generate weight gradients, the computation is performed using transposed activations. Therefore, if PPU 202 is performing backpropagation computation to generate weight gradients and input data 206 includes activations, group buffer 204 determines that the activations included in input data 206 should be transposed.

[0078] If group buffer 204 determines that the input data 206 should not be transposed, the method proceeds to step 406, where group buffer 204 transmits the input data 206 to PPU 202. If the input data 206 includes quantized data, group buffer 204 also transmits one or more scaling factors for generating the quantized data.

[0079] If group buffer 204 determines that the input data 206 should be transposed, the method proceeds to step 408, where group buffer 204 transposes the input data 206 or a portion thereof. For example, if the input data 206 includes activation gradients and weights, group buffer 204 transposes the weights but not the activation gradients. The transposition of the input data 206 is performed in a manner similar to that discussed above with respect to group buffer 204 and data path unit 218.

[0080] In some embodiments, group buffer 204 retrieves data to be transposed from unified buffer 210 and transmits the retrieved data to data path unit 218. Data path unit 218 transposes the data to generate a transposed data set.

[0081] In some embodiments, input data 206 includes multiple sets of input data. In step 404, group buffer 204 determines for each set of input data whether it should be transposed. For example, input data 206 may include a set of weights 214 and a set of gradients 216. Group buffer 204 determines that the weights 214 should be transposed and the gradients 216 should not be transposed. In step 408, group buffer 204 transposes the weights 214 without transposing the gradients 216.

[0082] In step 410, group buffer 204 transmits the input data 206, which includes the transposed data, to PPU 202. If the input data 206 includes quantized data, group buffer 204 also transmits one or more scaling factors for generating the quantized data.

[0083] In step 412, group buffer 204 receives output data 208 transmitted by PPU 202. The type of output data may vary depending on the type of input data 206 transmitted to PPU 202 and / or the type of neural network computation performed by PPU 202. If PPU 202 performs forward propagation computation, output data 208 includes output activations. If PPU 202 performs backpropagation computation, output data 208 includes gradients, such as weight gradients or input activation gradients. In some embodiments, output data 208 includes quantized data, such as quantized output activations, quantized weight gradients, and / or quantized input activation gradients. If output data 208 includes quantized data, output data 208 also includes one or more scaling factors for generating the quantized data.

[0084] In step 414, group buffer 204 determines whether to process output data 208. In some embodiments, the determination of whether to process output data 208 is based on the data type included in output data 208. For example, if output data 208 includes weights, output activation, or input activation gradients, then group buffer 204 determines that weights, output activation, or input activation gradients, in which case, should be stored in unified buffer 210 without further processing. If output data 208 includes weight gradients, then group buffer 204 determines that the weights stored in unified buffer 210 should be updated based on the weight gradients.

[0085] If group buffer 204 determines that output data 208 should not be processed, then in step 416, group buffer 204 stores output data 208 in unified buffer 210. In some embodiments, storing data in unified buffer 210 includes determining one or more memory addresses in unified buffer 210 corresponding to output data 208 and storing output data 208 based on one or more memory addresses.

[0086] If group buffer 204 determines that the output data 208 should be processed, then in step 418, group buffer 204 processes the output data 208. In some embodiments, group buffer 204 transmits the output data 208 to data path unit 218 for processing. Processing the output data 208 includes, for example, updating data stored in unified buffer 210 based on the output data 208.

[0087] Figure 5 This is a flowchart of method steps for performing neural network computation operations using quantized data, according to various embodiments. Although referenced... Figure 1 and Figure 2The system described herein is for reference only; however, those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of this invention.

[0088] like Figure 5 As shown, method 500 begins at step 502, where PPU 202 receives a first set of input data and a first set of scaling factors corresponding to the first set of input data from group buffer 204. Each scaling factor included in the first set of scaling factors is used to quantize a different subset of the input data included in the first set of input data. For example, the first set of input data may be a set of weights, input activations, output gradients, transposed weights, or transposed input activations. The type of input data is based on the expected input of the current neural network computation being performed by PPU 202. For example, if PPU 202 is performing a forward propagation computation, the first set of input data may be a set of weights or a set of input activations.

[0089] In step 504, PPU 202 stores the first input data set and the first scaling factor set in the first data buffer. The storage of the first input data set and the first scaling factor set is performed in a manner similar to that discussed above regarding data buffers 220(1) and 220(2). In some embodiments, PPU 202 identifies the first data buffer from two or more data buffers (e.g., data buffers 220(1) and 220(2)) included in PPU 202 based on the type of data included in the first input data set. For example, refer to... Figure 2 Data buffer 220(1) can be configured to store first type of input data, such as weights, while data buffer 220(2) can be configured to store second type of input data, such as activations. PPU 202 determines whether to store the first input data set in data buffer 220(1) or data buffer 220(2) based on whether the first input data set is first type of input data or second type of input data.

[0090] In some embodiments, the first input data set corresponds to a quantized multidimensional tensor. The multidimensional tensor is quantized by partitioning it into multiple matrices. A scaling factor is calculated for each matrix, and elements in the matrix are quantized based on this calculated scaling factor. Each scaling factor included in the first set of scaling factors corresponds to a quantization matrix in the quantized multidimensional tensor. The PPU 202 stores each scaling factor adjacent to its corresponding quantization matrix, such that the matrix values ​​and scaling factors can be read or loaded simultaneously. For example, the first data buffer can store data in multiple rows, and the PPU 202 can store each quantization matrix and its corresponding scaling factor in different rows. Furthermore, in some embodiments, the first data buffer includes multiple memory banks. In some embodiments, the PPU 202 stores each column of the quantization matrix in the same row of different memory banks. In other embodiments, the PPU 202 stores each row of the quantization matrix in the same row of different memory banks.

[0091] In step 506, PPU 202 receives a second set of input data and a second set of scaling factors corresponding to the second set of input data from group buffer 204. Each scaling factor included in the second set of scaling factors is used to quantize a different subset of input data contained in the second set of input data. The second set of input data can be, for example, a set of weights, input activations, output gradients, transposed weights, or transposed input activations. The type of input data in the second set of input data is based on the expected input of the current neural network computation being performed by PPU 202 and the type of input data in the first set of input data. For example, if PPU 202 is performing a forward propagation computation and the first set of input data is a set of weights, then the second set of input data is a set of input activations.

[0092] In step 508, PPU 202 stores the second input data set and the second scaling factor set in the second data buffer. The storage of the second input data set and the second scaling factor set is performed in a manner similar to that discussed above regarding data buffers 220(1) and 220(2). In some embodiments, PPU 202 identifies the second data buffer from two or more data buffers (e.g., data buffers 220(1) and 220(2)) included in PPU 202 based on the data type included in the second input data set. For example, refer to... Figure 2Data buffer 220(1) can be configured to store a first type of input data, such as weights, while data buffer 220(2) can be configured to store a second type of input data, such as activations. PPU 202 determines whether to store the second input data set in data buffer 220(1) or data buffer 220(2) based on whether the second input data set is the first type of input data or the second type of input data. In some embodiments, PPU 202 identifies data buffers not used to store the first input data set.

[0093] In some embodiments, the second input data set corresponds to a quantized multidimensional tensor, including each scaling factor in the second scaling factor set corresponding to a quantization matrix in the quantized multidimensional tensor. PPU 202 stores each scaling factor adjacent to the corresponding quantization matrix, such that the matrix value and the scaling factor can be read or loaded simultaneously. For example, the second data buffer can store data in multiple rows, and PPU 202 can store each quantization matrix and its corresponding scaling factor in different rows. Furthermore, in some embodiments, the second data buffer includes multiple memory banks. In some embodiments, PPU 202 stores each column of the quantization matrix in the same row of different memory banks. In other embodiments, PPU 202 stores each row of the quantization matrix in the same row of different memory banks.

[0094] In some embodiments, two or more of the first input data set, the first scaling factor set, the second input data set, and the second scaling factor set are included in the same input data set 206 received from the group memory 204. The PPU 202 can divide the input data 206 into two or more of the first input data set, the first scaling factor set, the second input data set, and the second scaling factor set.

[0095] In some embodiments, the second input data set includes quantized data, but the first input data set does not include quantized data. In such an embodiment, in step 502, PPU 202 does not receive the first scaling factor set, and in step 504, PPU 202 does not store the first scaling factor set. Furthermore, in some embodiments, the first input data set includes quantized data, but the second input data set does not include quantized data. In such an embodiment, in step 506, PPU 202 does not receive the second scaling factor set, and in step 508, PPU 202 does not store the second scaling factor set.

[0096] In some embodiments, steps 502-504 and / or steps 506-508 described above can be repeated for the same type of additional input data. The number of times steps 502-504 are performed may differ from the number of times steps 506-508 are performed. For example, PPU 202 may receive a single set of weights instead of multiple sets of input activations. Furthermore, steps 502-508 may be performed in a different order and / or in parallel. For example, steps 506-508 may be performed before or simultaneously with steps 502-504.

[0097] In step 510, PPU 202 performs one or more neural network computations using a first input data set, a second input data set, a first scaling factor set, and a second scaling factor set. The one or more neural network computations are performed in a manner similar to that discussed above regarding PPU 202, vector MAC unit 226, and accumulation unit 228. In some embodiments, the first input data set corresponds to a first multidimensional tensor, and the second input data set corresponds to a second multidimensional tensor. Performing one or more neural network computations includes performing matrix multiplication between matrices included in the first multidimensional tensor and matrices included in the second multidimensional tensor.

[0098] In some embodiments, PPU 202 reads a first row of data from a first data buffer and a second row of data from a second data buffer. The first row of data from the first data buffer corresponds to a first matrix included in the first input data set and a scaling factor corresponding to the first matrix. The second row of data from the second data buffer corresponds to a second matrix included in the second input data set and a scaling factor corresponding to the second matrix.

[0099] In some embodiments, PPU 202 reads a first row of data from a first memory group in a first data buffer and reads a second row of data from a second memory group in a second data buffer. The first row of data from the first memory group corresponds to a column of a first matrix included in the first input data set. The second row of data from the second memory group corresponds to a column of a second matrix included in the second input data set.

[0100] PPU 202 multiplies each element included in a data row from the first data buffer with the corresponding element included in a data row from the second data buffer. PPU 202 sums the products to generate a partial sum. Furthermore, it multiplies the scaling factor corresponding to the first matrix with the scaling factor corresponding to the second matrix to produce a scaling factor product. PPU 202 scales the partial sum by multiplying it by the scaling factor product. If the first input data set includes quantized data but the second input data set does not, PPU 202 scales the partial sum by multiplying it by the scaling factor corresponding to the first matrix. Similarly, if the second input data set includes quantized data but the first input data set does not, PPU 202 scales the partial sum by multiplying it by the scaling factor corresponding to the second matrix.

[0101] PPU 202 repeatedly performs multiplication and accumulation calculations on each row of data from the first and second memory sets to generate multiple partial sums. Each partial sum corresponds to a different element of the multidimensional output tensor. For each element of the output tensor, PPU 202 accumulates the partial sum corresponding to that element. After PPU 202 has completed the calculations on all data included in the first and second input data sets, the accumulated partial sums then form the complete output tensor.

[0102] In step 512, PPU 202 performs one or more post-processing operations on the output data to generate processed output data. These one or more post-processing operations are performed in a manner similar to that discussed above regarding PPU 202 and post-processing unit 230.

[0103] In some embodiments, PPU 202 scales the output data based on a per-layer scaling factor. Scaling the output data based on the per-layer scaling factor includes scaling each element of the output tensor using the per-layer scaling factor. In some embodiments, PPU 202 also receives the per-layer scaling factor in addition to or in combination with the first and / or second input data sets. In some embodiments, PPU 202 calculates the per-layer scaling factor based on the output data.

[0104] In some embodiments, PPU 202 processes the output data using a nonlinear function. Processing the output data using the nonlinear function includes applying the nonlinear function to each element of the output tensor. In some embodiments, PPU 202 first scales the output data based on a per-layer scaling factor to generate scaled output data, and then processes the scaled output data using the nonlinear function.

[0105] In some embodiments, PPU 202 quantizes output data to generate quantized output data. Quantizing the output data includes quantizing each element of the output tensor to an N-bit integer based on a scaling factor. For example, the quantization of the elements is performed using equation (1) discussed above. Different elements of the output tensor may be quantized based on different scaling factors. In some embodiments, PPU 202 divides the output tensor into multiple matrices and quantizes each matrix using a different scaling factor. Furthermore, in some embodiments, PPU 202 calculates a scaling factor for quantizing the matrix elements for each matrix.

[0106] In some embodiments, PPU 202 first scales the output data based on a scaling factor for each layer, and then quantizes the scaled output data. In other embodiments, PPU 202 scales the output data and processes the output data using the nonlinear function before quantizing the output data.

[0107] In some embodiments, when performing neural network training computations, the PPU 202 selects the bit width used to quantize the output tensor based on the current epoch (iteration) of the neural network training. One advantage of using different bit widths for different iterations of neural network training is that lower bit widths can be used during earlier iterations of neural network training to improve energy efficiency and processing performance. Higher bit widths can be used in later iterations of neural network training, where higher-precision computations improve the accuracy of the neural network model.

[0108] In step 514, PPU 202 transmits the processed output data 208 to group buffer 204. Transmitting the processed output data 208 includes transmitting the quantized output tensor and different scaling factors for the elements used to quantize the output tensor.

[0109] In summary, the disclosed technique enables efficient neural network inference and training computations using quantized input data. To generate a quantized tensor, a multidimensional tensor is divided into multiple submatrices along its multiple dimensions. Each submatrix is ​​quantized based on a corresponding calculated scaling factor. The quantized submatrices are stored together with their corresponding scaling factors. During neural network computation, each quantized submatrix and its corresponding scaling factor are read from memory. The neural network computation is performed using the quantized submatrices and their corresponding scaling factors. When performing neural network computation using the transpose of the quantized submatrices, the same corresponding scaling factor is read from memory using the transposed quantized submatrix. Then, the neural network computation is performed using the transposed quantized submatrix and its corresponding scaling factor. The output data generated from the neural network computation is accumulated to form an output tensor. The output tensor is quantized by dividing it into multiple matrices and calculating a scaling factor for each matrix. Each matrix is ​​quantized using the calculated scaling factor to generate the quantized output tensor.

[0110] At least one technical advantage of the disclosed technique over existing techniques is that neural network training operations can be performed more efficiently. In this regard, the disclosed technique applies different scaling factors to different submatrices along multiple dimensions of a tensor, rather than to different vectors along a single dimension of the tensor. Therefore, the same scaling factor can be applied to submatrices within a matrix and to corresponding submatrices within the corresponding transpose matrix. Consequently, compared to existing methods, performing computations using the transpose matrix requires fewer read operations, thereby reducing processing time when training neural networks. These technical advantages provide one or more technical improvements superior to existing methods.

[0111] 1. In some embodiments, a computer-implemented method for processing a neural network, the method comprising: receiving a first quantization matrix corresponding to a portion of a multidimensional input tensor and having been quantized based on a first scaling factor; and performing one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to a first portion of a multidimensional output tensor.

[0112] 2. The computer-executed method according to Clause 1 further includes: receiving a second quantization matrix, the second quantization matrix corresponding to a portion of a second multidimensional input tensor and having been quantized based on a second scaling factor; wherein performing the one or more computational operations further includes: using the second quantization matrix and the second scaling factor to generate the one or more data values.

[0113] 3. The computer execution method according to clause 1 or 2, wherein performing the one or more computational operations comprises: calculating one or more dot product values ​​using the first quantization matrix and the second quantization matrix.

[0114] 4. The computer-executed method according to any one of clauses 1-3, wherein performing the one or more computational operations comprises: scaling each of the dot product values ​​included in the one or more dot product values ​​using the first scaling factor and the second scaling factor.

[0115] 5. The computer-executed method according to any one of clauses 1-4, wherein the first quantization matrix includes a transpose matrix.

[0116] 6. The computer-executed method according to any one of clauses 1-5, wherein the first quantization matrix is ​​included in a plurality of quantization matrices, wherein each quantization matrix included in the plurality of quantization matrices corresponds to a different portion of the multidimensional input tensor and has been quantized using a different scaling factor.

[0117] 7. The computer-executed method according to any one of claims 1-6, wherein performing the one or more computational operations further comprises: for each of the plurality of quantization matrices, using the quantization matrix and an associated scaling factor to generate one or more associated data values, the one or more associated data values ​​corresponding to different portions of the multidimensional output tensor.

[0118] 8. The computer-executed method according to any one of clauses 1-7, further comprising: scaling the multidimensional output tensor based on a second scaling factor associated with the multidimensional output tensor.

[0119] 9. The computer-executed method according to any one of claims 1-8, further comprising: quantizing the multidimensional output tensor by subdividing the multidimensional output tensor into a plurality of matrices and quantizing each of the matrices included in the plurality of matrices.

[0120] 10. The computer-executed method according to any one of clauses 1-9, further comprising: quantizing the multidimensional output tensor by: calculating a corresponding scaling factor for each matrix included in a plurality of matrices comprising the multidimensional output tensor based on one or more data values ​​included in the matrix; and quantizing each matrix included in the plurality of matrices based on the corresponding scaling factor.

[0121] 11. In some embodiments, one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: receiving a first quantization matrix corresponding to a portion of a multidimensional input tensor and already quantized based on a first scaling factor; and performing one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to a first portion of the multidimensional output tensor.

[0122] 12. One or more non-transitory computer-readable media according to Clause 11, further comprising: receiving a second quantization matrix corresponding to a portion of a second multidimensional input tensor and having been quantized based on a second scaling factor; wherein performing the one or more computational operations further comprises: using the second quantization matrix and the second scaling factor to generate the one or more data values.

[0123] 13. One or more non-transitory computer-readable media as described in clause 11 or 12, wherein performing the one or more computational operations comprises: calculating a first dot product between a first vector included in the first quantization matrix and a second vector included in the second quantization matrix.

[0124] 14. One or more non-transitory computer-readable media according to any one of clauses 11-13, wherein performing the one or more computational operations comprises scaling a first dot product value based on at least one of the first scaling factor or the second scaling factor.

[0125] 15. One or more non-transitory computer-readable media according to any one of clauses 11-14, further comprising: receiving the first scaling factor and storing the first quantization matrix and the first scaling factor in a data buffer.

[0126] 16. One or more non-transitory computer-readable media according to any one of clauses 11-15, wherein performing the one or more computational operations comprises: reading from the data buffer a first set of input values ​​included in the first quantization matrix; and reading from the data buffer the first scaling factor, wherein the first set of input values ​​is also used to perform the one or more computational operations.

[0127] 17. One or more non-transitory computer-readable media according to any one of clauses 11-16, wherein the data buffer comprises a plurality of rows, and wherein storing the first quantization matrix and the first scaling factor in the data buffer comprises: storing the first quantization matrix and the first scaling factor in a first row included in the plurality of rows.

[0128] 18. One or more non-transitory computer-readable media according to any one of clauses 11-17, wherein the data buffer comprises a plurality of memory groups, wherein storing the first quantization matrix in the data buffer comprises: for each column included in the first quantization matrix, storing the column in a different memory group included in the plurality of memory groups.

[0129] 19. One or more non-transitory computer-readable media according to any one of clauses 11-18, wherein the first quantization matrix is ​​included in a plurality of quantization matrices, wherein each quantization matrix included in the plurality of quantization matrices corresponds to a different portion of the multidimensional input tensor and has been quantized using a different scaling factor, and further comprising: for each quantization matrix included in the plurality of quantization matrices, receiving the quantization matrix and an associated scaling factor, and storing the quantization matrix and the associated scaling factor in different rows of a data buffer.

[0130] 20. In some embodiments, a system includes: one or more memories storing instructions; and one or more processors coupled to the one or more memories, and performing the following steps when executing the instructions: receiving a first quantization matrix corresponding to a portion of a multidimensional input tensor and having been quantized based on a first scaling factor; and performing one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to a first portion of a multidimensional output tensor.

[0131] Any and all combinations of any claim element recited in any way in any claim and / or any element described in this application fall within the scope of this invention and protection.

[0132] Various embodiments have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0133] Aspects of this embodiment may be embodied as a system, method, or computer program product. Therefore, aspects of this disclosure may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all collectively referred to herein as a “module,” “system,” or “computer.” Furthermore, any hardware and / or software technology, processing, function, component, engine, module, or system described in this disclosure may be implemented as a circuit or group of circuits. Additionally, aspects of this disclosure may take the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0134] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium includes, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0135] Aspects of this disclosure have been described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. These instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0136] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code comprising one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative embodiments, the functions indicated in the blocks may not occur in the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or a combination of dedicated hardware and computer instructions.

[0137] While the foregoing description is directed to embodiments of this disclosure, other and further embodiments of this disclosure may be designed without departing from its essential scope, as defined by the appended claims.

Claims

1. A computer-implemented method for processing neural networks, the method comprising: When performing neural network training: - Receive a first quantization matrix, the first quantization matrix corresponding to a portion of a multidimensional input tensor and having been quantized based on a first scaling factor, wherein the portion of the multidimensional input tensor is a first matrix partitioned from the multidimensional input tensor, the number of rows and columns of the first matrix corresponding to the size of a vector multiplication accumulation MAC unit; and - Perform one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to the first part of the multidimensional output tensor; The trained neural network is used to perform graphics and video processing operations to output pixels.

2. The computer-implemented method according to claim 1 further includes: Receive a second quantization matrix, which corresponds to a portion of a second multidimensional input tensor and has been quantized based on a second scaling factor; Performing the one or more calculation operations further includes: using the second quantization matrix and the second scaling factor to generate the one or more data values.

3. The computer-implemented method according to claim 2, wherein, Performing the one or more computational operations includes: calculating one or more dot product values ​​using the first quantization matrix and the second quantization matrix.

4. The computer-implemented method according to claim 2, wherein, Performing the one or more calculation operations includes scaling each of the one or more dot product values ​​using the first scaling factor and the second scaling factor.

5. The computer-implemented method according to claim 1, wherein, The first quantization matrix includes the transpose matrix.

6. The computer-implemented method according to claim 1, wherein, The first quantization matrix is ​​included in a plurality of quantization matrices, wherein each of the plurality of quantization matrices corresponds to a different part of the multidimensional input tensor and has been quantized using a different scaling factor.

7. The computer-implemented method according to claim 6, wherein, Performing the one or more computational operations further includes: for each of the plurality of quantization matrices, using the quantization matrix and an associated scaling factor to generate one or more associated data values, the one or more associated data values ​​corresponding to different portions of the multidimensional output tensor.

8. The computer-implemented method according to claim 1, further comprising: The multidimensional output tensor is scaled based on a second scaling factor associated with it.

9. The computer-implemented method according to claim 1, further comprising: The multidimensional output tensor is quantized by subdividing it into multiple square matrices and quantizing each square matrix included in the multiple square matrices.

10. The computer-implemented method according to claim 1, further comprising: The multidimensional output tensor is quantized in the following way: For each of the multiple square matrices including the multidimensional output tensor, a corresponding scaling factor is calculated based on one or more data values ​​included in the square matrix; as well as The quantization based on the corresponding scaling factor is included in each of the plurality of square matrices.

11. One or more non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps: When performing neural network training: - Receive a first quantization matrix, which corresponds to a portion of the multidimensional input tensor and has been quantized based on a first scaling factor, wherein... The portion of the multidimensional input tensor is a first square matrix partitioned from the multidimensional input tensor, wherein the number of rows and columns of the first square matrix corresponds to the size of the vector multiplication accumulation MAC unit; as well as - Perform one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to the first part of the multidimensional output tensor; The trained neural network is used to perform graphics and video processing operations to output pixels.

12. The non-transitory computer-readable medium of claim 11, further comprising: Receive a second quantization matrix, which corresponds to a portion of a second multidimensional input tensor and has been quantized based on a second scaling factor; Performing the one or more calculation operations further includes: using the second quantization matrix and the second scaling factor to generate the one or more data values.

13. One or more non-transitory computer-readable media according to claim 12, wherein, Performing the one or more computational operations includes: calculating a first dot product between a first vector included in the first quantization matrix and a second vector included in the second quantization matrix.

14. One or more non-transitory computer-readable media according to claim 12, wherein performing the one or more computational operations comprises: The first dot product value is scaled based on at least one of the first scaling factor or the second scaling factor.

15. The non-transitory computer-readable medium of claim 11, further comprising: The first scaling factor is received and the first quantization matrix and the first scaling factor are stored in the data buffer.

16. One or more non-transitory computer-readable media according to claim 15, wherein performing the one or more computational operations comprises: Read the first set of input values ​​included in the first quantization matrix from the data buffer; And read the first scaling factor from the data buffer, wherein the first set of input values ​​is also used to perform the one or more calculation operations.

17. One or more non-transitory computer-readable media according to claim 15, wherein, The data buffer includes multiple rows, and storing the first quantization matrix and the first scaling factor in the data buffer includes storing the first quantization matrix and the first scaling factor in a first row included in the multiple rows.

18. One or more non-transitory computer-readable media according to claim 15, wherein, The data buffer includes multiple memory groups, wherein storing the first quantization matrix in the data buffer includes: for each column included in the first quantization matrix, storing the column in a different memory group included in the multiple memory groups.

19. One or more non-transitory computer-readable media according to claim 11, wherein, The first quantization matrix is ​​included in a plurality of quantization matrices, wherein each quantization matrix included in the plurality of quantization matrices corresponds to a different part of the multidimensional input tensor and has been quantized using different scaling factors, and further includes: for each quantization matrix included in the plurality of quantization matrices, receiving the quantization matrix and the associated scaling factor, and storing the quantization matrix and the associated scaling factor in different rows of a data buffer.

20. A system comprising: One or more memories for storing instructions; and One or more processors, coupled to the one or more memories, perform the following steps when executing the instructions: - When performing neural network training: --Receive a first quantization matrix, the first quantization matrix corresponding to a portion of a multidimensional input tensor and already quantized based on a first scaling factor, wherein the portion of the multidimensional input tensor is a first matrix partitioned from the multidimensional input tensor, the number of rows and columns of the first matrix corresponding to the size of a vector multiplication accumulation MAC unit; and -- Perform one or more computational operations using the first quantization matrix and the first scaling factor to generate one or more data values ​​corresponding to the first part of the multidimensional output tensor; - Use the trained neural network to perform graphics and video processing operations to output pixels.

Images