Utilizing data sparsity at machine learning hardware accelerators

Abstract

Methods and systems, including computer readable media, are described for utilizing data sparsity during computation of neural networks implemented on hardware accelerators. Using a system controller, a compressed sparse set of parameters is derived from the parameter tensor, and a mapping vector is generated based on the compressed sparse set of parameters. When a system processes an opcode in an instruction indicating sparsity of a parameter tensor, an input vector is obtained from a first memory of a hardware accelerator, and a compressed sparse parameter and a mapping vector are retrieved from a second memory of the hardware accelerator. The input vector is processed through a layer of the neural network using the mapping vector and the compressed set of sparse parameters.

Description

Background Art

[0001] The present specification generally relates to using hardware integrated circuits to perform group convolutions of convolutional neural networks.

[0002] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, such as a classification, for a received input. In addition to the output layer, some neural networks also include one or more hidden layers. Some neural networks may be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform a variety of tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

[0003] A neural network layer can have a corresponding set of parameters or weights. The weights are used to process inputs (e.g., a batch of inputs) through a neural network layer to generate a corresponding output of the layer to compute a neural network inference. A batch of inputs and a set of kernels can be represented as a tensor, i.e., a multidimensional array, of inputs and weights. A hardware accelerator is a dedicated integrated circuit for implementing a neural network. The circuit includes memory whose locations correspond to elements of the tensor that can be traversed or accessed using the control logic of the circuit. Summary of the invention

[0004] This document describes an improved integrated circuit architecture for a hardware accelerator, and corresponding techniques for processing an input vector using a mapping vector and a compressed sparse parameter set for a neural network layer. Each of the mapping vector and the compressed sparse parameter set can be generated based on an operation code ("opcode") that indicates a uniform sparsity format of multiple parameter tensors. The parameter tensors are associated with a neural network layer of an artificial neural network such as a CNN. The disclosed techniques can be used to accelerate tensor operations supporting neural network computations that involve processing inputs of the input vector through one or more of the neural network layers.

[0005] One aspect of the subject matter described in this specification can be embodied in a computer-implemented method involving a neural network implemented on a hardware accelerator. The method includes: deriving a compressed sparse parameter set from a parameter tensor; generating a mapping vector based on the compressed sparse parameter set; processing an instruction that instructs to perform a sparse computation using the compressed sparse parameters based on the sparsity of the parameter tensor; obtaining, based on the instruction: i) an input vector from a first memory of the hardware accelerator and ii) compressed sparse parameters from a second memory of the hardware accelerator; and performing the sparse computation using the mapping vector and the compressed sparse parameter set to process the input vector through a layer of the neural network.

[0006] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, processing an input vector through a layer of a neural network includes performing a dot product matrix multiplication operation between an input of the input vector and a corresponding weight value in a compressed sparse parameter set.

[0007] In some implementations, a dot product matrix multiplication operation is performed at one or more multiplication cores of the hardware accelerator based on a corresponding bit value of each bit in the mapping vector. In some implementations, the method further includes: accessing hardware selection logic coupled to a first memory and a second memory of the hardware accelerator; and using the hardware selection logic, selecting a particular input of the input vector and a corresponding weight value in the compressed sparse parameter set based on the corresponding bit value of the bit in the mapping vector.

[0008] Deriving the compressed sparse parameter set may include generating a modified parameter tensor that includes only non-zero elements along a particular dimension of the parameter tensor. Generating the modified parameter tensor may include: for a particular column dimension of the parameter tensor: generating a compressed representation of the column dimension based on the non-zero elements of the column dimension; and concatenating each non-zero element in the compressed representation of the column dimension.

[0009] In some implementations, generating the modified parameter tensor includes: preserving the corresponding dimensional position of each non-zero element in the parameter tensor before generating the modified parameter tensor. The parameter tensor may include multiple dimensions; and the opcode in the instruction indicates the sparsity of a particular dimension in the multiple dimensions. The hardware accelerator is operable to process a multi-dimensional parameter tensor; and the opcode in the instruction may indicate uniform sparsity across each of the multi-dimensional parameter tensors.

[0010] The first memory may be a scratchpad memory of the hardware accelerator and is configured to store inputs and activations processed at the neural network layer. The second memory may include a single instruction multiple data (SIMD) register, and the method includes: storing the mapping vector at a first address of the SIMD register; and storing the compressed sparse parameter set at a second different address of the SIMD register.

[0011] Another aspect of the subject matter described in this specification can be embodied in a computer-implemented method performed using a hardware accelerator that implements a neural network including a plurality of neural network layers. The method includes receiving instructions for a compute chiplet of the hardware accelerator.

[0012] The instruction is executable at a compute chiplet to cause operations to be performed, the operations comprising: identifying an opcode in the instruction indicating sparsity of a parameter tensor; loading a compressed sparse parameter set based on weight values ​​derived from a parameter tensor of weights of a layer of a specified neural network; and loading a mapping vector generated based on the compressed sparse parameter set.

[0013] The operations include obtaining, based on the opcode, i) an input vector from a first memory of the hardware accelerator and ii) a compressed sparse parameter set from a second memory of the hardware accelerator. The operations further include processing the input vector through a layer of a neural network using the compressed sparse parameter set based on the mapping vector.

[0014] Other implementations of this and other aspects include corresponding systems, devices, and computer programs configured to perform the actions of these methods, encoded on computer storage devices. The system of one or more computers may be so configured due to software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions during operation. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform actions.

[0015] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages. Techniques for exploiting sparsity in data processed for machine learning computations are described. When processing input images using, for example, a CNN machine learning model implemented on a computing device such as a tablet or smartphone, certain hardware and computational efficiencies are achieved by utilizing compressed sparse parameters having only non-zero weight values.

[0016] Sparsity is exploited to achieve computational efficiency when accelerating the execution of an artificial neural network by generating compressed sparse parameters and corresponding mapping vectors. The system detects upcoming sparsity patterns in a data set to be processed at a neural network layer and generates a set of compressed sparse parameters that include only non-zero values. The mapping vector maps discrete inputs of an input vector to non-zero values ​​of compressed sparse parameters, which allows for simplified processing of data sets by utilizing a specific hardware architecture of an application specific integrated circuit that accelerates the execution of an artificial neural network.

[0017] Multiplication operations involving zero-valued operands are generally viewed as wasted computation cycles. By using at least compressed sparse parameters to process neural network inputs that only have non-zero values, a machine learning system can reduce its total amount of computation operations. This reduction is achieved by removing zero values ​​from the weight values ​​of the parameter tensors processed for the neural network layers. The reduced amount of computation operations results in a corresponding reduction in power consumption and resource requirements (e.g., memory allocation and processor cycles).

[0018] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an example computing system for implementing a neural network machine learning model.

[0020] Figure 2 An example parameter tensor with K-in-N sparsity is shown.

[0021] Figure 3 A first example architecture for processing compressed sparse parameters is shown.

[0022] Figure 4 A second example architecture for processing compressed sparse parameters is shown.

[0023] Figure 5 An example processing pipeline for routing input obtained from a memory location to one or more compute cores is shown.

[0024] Figure 6 is an example procedure for exploiting data sparsity at a machine learning hardware accelerator.

[0025] Figure 7 Examples of input tensors, parameter tensors, and output tensors are illustrated.

[0026] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0027] Figure 1 1 is a block diagram of an example computing system 100 for implementing a neural network model at a hardware integrated circuit such as a machine learning hardware accelerator. The computing system 100 includes one or more computing chipsets (tiles) 101, a host 120, and a higher-level controller 125 ("controller 125"). As described in more detail below, the host 120 and the controller 125 cooperate to provide data sets and instructions to one or more computing chipsets 101 of the system 100.

[0028] In some implementations, the host 120 and the controller 125 are the same device. The host 120 and the controller 125 may also perform distinct functions but be integrated in a single device package. For example, the host 120 and the controller 125 may form a central processing unit (CPU) that interacts or collaborates with a hardware accelerator including a plurality of computing chiplets 101. In some implementations, the host 120, the controller 125, and the plurality of computing chiplets 101 are included or formed on a single integrated circuit die. For example, the host 120, the controller 125, and the plurality of computing chiplets 101 may form a dedicated system on chip (SoC) that is optimized for executing neural network models for processing machine learning workloads.

[0029] Each compute chiplet 101 typically includes a controller 103 that provides one or more control signals 105 to cause the input (or activation) of the input vector 102 to be stored at or accessed from a memory location of a first memory 108 ("memory 108"). Similarly, the controller 103 may also provide one or more control signals 105 to cause the weights (or parameters) of the matrix structure of weights 104 to be stored at or accessed from a memory location of a second memory 110 ("memory 110"). In some implementations, the input vector 102 is obtained from an input tensor, and the matrix structure of weights 103 is obtained from a parameter tensor. Each of the input tensor and the parameter tensor can be a multi-dimensional data structure, such as a multi-dimensional matrix or tensor. This is described below with reference to Figure 7 Describe in more detail.

[0030] Each memory location of memories 108, 110 can be identified by a corresponding memory address. Each of memories 108, 110 can be implemented as a series of banks, cells, or any other related storage medium or device. Each of memories 108, 110 can include one or more registers, buffers, or both. In general, controller 103 arbitrates access to each of memories 108, 110. In some implementations, inputs or activations are stored at memory 108, memory 110, or both; and weights are stored at memory 110, memory 108, or both. For example, inputs and weights can be transferred between memory 108 and memory 110 to facilitate certain neural network calculations.

[0031] Each compute chiplet 101 also includes an input activation bus 106, an output activation bus 107, and a compute unit 112 including a multiply accumulate cell (MAC) 114 a / b / c. The controller 103 may generate a control signal 105 to obtain operands stored at a memory of the compute chiplet 101. For example, the controller 103 may generate a control signal 105 to obtain: i) an example input vector 102 stored at a memory 108 and ii) a weight 104 stored at a memory 110. Each input obtained from the memory 108 is provided to the input activation bus 106 for routing (e.g., direct routing) to a compute cell 114 a / b / c in the compute unit 112. Similarly, each weight obtained from the memory 110 is routed to a cell 114 a / b / c of ​​the compute unit 104.

[0032] As described below, each core 114a / b / c performs a computation that produces a partial sum or accumulated value to generate an output for a given neural network layer. An activation function may be applied to a set of outputs to generate a set of output activations for a neural network layer. In some implementations, the outputs or output activations are routed via an output activation bus 107 for storage and / or transmission. For example, a set of output activations may be transmitted from a first computing chiplet 101 to a second, different computing chiplet 101 for processing at the second computing chiplet 101 as input activations for a different layer of a neural network.

[0033] In general, each compute chiplet 101 and system 100 may include additional hardware structures to perform computations associated with multi-dimensional data structures such as tensors, matrices, and / or data arrays. In some implementations, input vectors (or tensors) 102 inputs and weights 104 of parameter tensors may be preloaded into the memory 108, 110 of the compute chiplet 101. The inputs and weights are received as a set of data values ​​that arrive at a particular compute chiplet 101 from a host 120 (e.g., an external host) via a host interface or from a higher level control such as a controller 125.

[0034] Each of the compute chiplets 101 and controller 103 may include one or more processors, processing devices, and various types of memory. In some implementations, the processors of the compute chiplets 101 and controller 103 include one or more devices, such as a microprocessor or central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a combination of different processors. Each of the compute chiplets 101 and controller 103 may also include other computing and storage resources, such as buffers, registers, control circuit systems, etc. These resources cooperate to provide additional processing options for performing one or more determinations and calculations described in this specification.

[0035] In some implementations, the processing unit of the controller 103 executes programmed instructions stored in the memory to cause the controller 103 and the computing chiplet 101 to perform one or more functions described in this specification. The memory of the controller 103 may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage medium may include a solid-state memory, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information or instructions.

[0036] The system 100 receives instructions defining a particular computing operation to be performed by the compute chiplet 101. In some implementations, the host may generate a set of compressed parameters (CSP) and corresponding mapping vectors, such as a non-zero mapping (NZM), for a given operation. For example, the host 120 may send the compressed parameters to the compute chiplet 101 via a host interface for further processing at the chiplet. The controller 103 may execute programmed instructions to analyze a data stream associated with received weights and inputs including compressed parameters and corresponding mapping vectors.

[0037] The controller 103 causes the input and weights of the data stream to be stored at the compute chiplet 101. For example, the controller 103 may store the mapping vectors and the compressed sparse parameters in a memory of the compute chiplet 101. This is described in more detail below. The controller 102 may also analyze the input data stream to detect an operation code ("opcode"). Based on the opcode, the controller 102 may activate dedicated data path logic associated with one or more compute cores 114a / b / c to perform sparse computations using the compressed sparse parameters and corresponding mapping vectors. As used in this document, sparse computations include neural network computations performed on a neural network layer using non-zero weight values ​​in a compressed sparse parameter set generated from a weight set of the neural network layer.

[0038] In some implementations, the opcode indicates the sparsity of one or more parameter tensors based on the values ​​of K and N (described below). The controller 103 detects the opcode including any relevant tensor sparsity information, uses local read logic to obtain compressed parameters from chiplet memory based on the opcode, and wires or routes these compressed parameters to the MAC 114 a / b / c of ​​the compute chiplet 101.

[0039] As described in detail below, the controller 103 can also analyze the example data stream and generate a compressed set of sparse parameters and a corresponding mapping vector that maps the discrete input of the input vector to the compressed sparse parameters based on the analysis. To the extent that operations and / or processes for generating compressed sparse parameters and corresponding mapping vectors are described with reference to the controller 103, each of these operations and processes can also be performed by the host 120, the controller 125, or both.

[0040] In some implementations, performing some (or all) of the operations at the host 120, such as analyzing tensor indices, performing direct memory access (DMA) operations to read address space in system memory (e.g., DRAM) to obtain input and weight values, generating compressed sparse parameters, and generating corresponding mapping vectors, will allow for reduced processing time at each compute chiplet 101 and increased data throughput at the system 100. For example, performing these operations at the host 120 using the controller 125 allows for sending already compressed parameter sets to a given chiplet compute 101, which reduces the size and amount of data that needs to be routed at the system 100.

[0041] Figure 2 An example parameter tensor 200 with K-in-N sparsity is shown, which can represent a uniformly sparse format exhibited by a sparse tensor. In general, for K-in-N sparsity, for every next N elements along a dimension (e.g., the innermost dimension) of the tensor, K elements are non-zero.

[0042] One or more opcodes may indicate or specify a sparsity property of one or more parameter tensors as well as sparsity along a particular column (or row) dimension of a given tensor. For example, an opcode in a single instruction received at a compute chiplet 101 may specify K-in-N sparsity of a parameter tensor 200, including K-in-N sparsity of each column 202 or row 204 of the parameter tensor 200. In some implementations, the tensor sparsity information specified by the opcode is based on the structure or configuration of the instruction set used at the system 100.

[0043] exist Figure 2In the example of , K indicates one or more non-zero values, and N is the number of elements of a given parameter tensor 200. In some examples, N is the number of elements of a given row or column of the parameter tensor. Each of K and N is an integer. N can be greater than or equal to one, and K can be greater than or equal to zero. K-in-N sparsity can be a ratio or other numerical value assigned to or communicated as a sparsity parameter.

[0044] The sparsity parameter characterizes a sparsity property or sparsity measure in a data set or tensor 200. For example, the sparsity parameter can represent a compression ratio for a given {K, N} pair and is equal to K / N, such that if K=2 and N=4, the compression ratio is 50%. The system 100 can support situations where the parameter is compressed in one (or more) dimensions, such as along a column dimension corresponding to columns 202. For this particular type of reduction operation, columns 202 can be described as reduction dimensions or inner product dimensions. In some implementations, the sparsity in the data set is based on one or more sparsity patterns detectable during a training phase of a neural network model, a deployment phase of a neural network model, or both.

[0045] The sparsity pattern can be evenly distributed among a machine learning dataset, such as parameter tensors 200 processed during the training and deployment phases of model execution. The uniformity of the sparsity pattern allows for a certain measure of predictability that can be exploited to achieve accelerated efficiency of the neural network model. For example, and as explained below, an evenly distributed sparsity pattern can allow for the prediction, inference, or otherwise detection of upcoming zero or non-zero weight value patterns (e.g., sparsity properties). In some implementations, each of the controllers 103, 125 can be configured to learn, explore, and exploit different pattern options to achieve additional efficiency and optimization in model execution.

[0046] exist Figure 2 In the example of , one or more opcodes received at the compute chiplet 101 may indicate that each of the columns 202 and rows 204 include K-in-N sparsity ½, where K=4 and N=8. The controller 103 determines the value of the sparsity parameter based on the logical expression: . In this example, controller 103 may assign a value of ½ to the corresponding sparsity parameter for each of columns 202 and 204. Relatedly, the opcode received at compute chiplet 101 may also specify that row 206 - which may also be a column - includes a K-in-N sparsity of 5 / 8, where K=5 and N=8. In some implementations, K for a given K-in-N sparsity is determined based on the hardware layout of the compute chiplet 101. For example, K may be determined based on the amount of MAC circuits in the hardware compute core of the compute unit 112 at a given compute chiplet 101.

[0047] Figure 3 A first example architecture 300 for processing parameter tensors to generate compressed sparse parameters is shown, and Figure 4 A second example architecture 400 for processing parameter tensors to generate compressed sparse parameters is shown. In view of the similarities between architectures 300 and 400, the following paragraphs describe them simultaneously. Figure 3 and Figure 4 Each one of them.

[0048] The controller 103 processes the opcode and triggers one or more operations to exploit the sparsity in the data set for machine learning workloads. The controller 103 can process the opcode to identify or determine, for a given parameter tensor, a corresponding sparsity metric (e.g., a sparsity parameter value) of the parameter tensor (such as a row of the tensor or a column of the tensor). The operation can also involve analysis of the parameter tensor. In some implementations, the controller 103 triggers an operation to exploit the sparsity of the parameter tensor in response to determining that the sparsity parameter value representing the metric sparsity of the parameter tensor exceeds a threshold parameter value.

[0049] Based on the opcode and any associated sparsity threshold comparison, the controller 103 triggers a determination of whether the weight value of the parameter tensor has a zero value or a non-zero value. For example, the controller 103 can analyze the discrete weight values ​​of the parameter tensor to detect a non-zero weight value. In response to detecting a non-zero weight value (e.g., the non-zero weight value can be indicated as K), the controller 103 then uses the non-zero weight value to generate a compressed parameter set or a compressed parameter grouping.

[0050] In some implementations, the controller 103 extracts the detected non-zero weight values ​​and generates a compressed parameter set using the extracted weights. In some other implementations, rather than extracting weight values, the controller 103 associates the detected non-zero weight values ​​with a compressed parameter set, such as a compressed parameter set previously generated by the host 120 and then passed to the controller 103 at the compute chiplet 101 through an example host interface. The controller 103 may also generate a compressed parameter packet using a combination of extraction and association.

[0051] The controller 103 maps each detected non-zero weight value to a mapping vector 302, 402, which can be represented as a bit vector, a bitmap, or other related data structure for indicating the correlation or mapping between different data items. In some implementations, the controller 103 determines the mapping of the mapping vector with reference to the corresponding input vector 304, 404. For example, the mapping vector 302, 402 maps the discrete input of the input vector 304, 404 to the non-zero value of the compressed sparse parameter set 305, 405, respectively.

[0052] In some implementations, the mapping vector is a non-zero bitmap NZM identified as a parameter. The example CSP may correspond to a modified parameter tensor derived for an original, unmodified parameter tensor, and the mapping is configured to preserve the corresponding dimensional position of each non-zero element in the original, unmodified parameter tensor before generating the modified parameter tensor. For example, the mapping vector may have the same dimensions as the original matrix for which the mapping vector is determined, but the mapping vector has a 1-bit data type, which is i) set to "1" for non-zero elements (e.g., non-zero weight values) in that position in the original matrix, or ii) set to "0" for zero elements (e.g., zero weight values) in that position in the original matrix.

[0053] Individual inputs of the input vector 304, 304 may be represented as {a0, a1, a2, a3, aN}, while individual weight values ​​of the parameter tensor may be represented as {w0, w1, w2, w3, wN}. The mapping vector 302, 402 uses a control value, such as a binary value, to map individual inputs (e.g., a0, a1, a2, etc.) of the input vector 304, 404 to non-zero weights in the set of compressed sparse parameters 305, 405. The compute chiplet 101 includes selection logic 314 for selecting individual inputs of the input vector 304, 404 with reference to the non-zero weights in the set of compressed sparse parameters 305, 405. The selection logic 314 references the mapping vector to align its extraction of the input in the input vector with the corresponding non-zero weight value in the set of compressed sparse parameters.

[0054] In some implementations, the selection logic 314 is implemented in hardware, software, or both. For example, the controller 103 can access hardware selection logic 314 coupled to the first memory 108 and the second memory 110 of the hardware accelerator. The controller 103 can use the selection logic 314 to select a particular input of the input vector and a corresponding weight value in the compressed sparse parameter set based on the corresponding bit values ​​of the bits in the mapping vector.

[0055] The controller 103 can generate a mapping vector that maps individual inputs of a multidimensional (3D) input tensor to non-zero weights in a multidimensional (3D) compressed sparse parameter tensor. In some implementations, for a given multidimensional tensor, the compute chiplet 101 is configured such that different cores or core groupings in the compute unit 112 are assigned to operate on different columns or dimensions of the parameter tensor / weight matrix. Therefore, the compute chiplet 101 can generate different bitmaps or mapping vectors for each core or each core grouping. In this way, each compute chiplet 101 can include corresponding selection logic uniquely configured for each core, each core grouping, or both.

[0056] The controller 103 generates control signals to store the compressed set of sparse parameters and the corresponding mapping vectors in a memory location at the compute chiplet 101. For example, the memory 110 may include single instruction multiple data (SIMD) registers, each SIMD register being configured to store the mapping vectors 302, 402, respectively, at a corresponding first address of the SIMD register 310. Similarly, the SIMD registers may also store the compressed set of sparse parameters 305, 405 at a corresponding second different address of the SIMD register 310.

[0057] The SIMD register 310 may include parallel cores for computing multiple dot products in parallel. The dot product computation may be performed using numeric formats or data types (dt) such as INT8, BFLOAT, HALF, and INT16. In some implementations, one or more data fields of an opcode and / or instruction received at the compute chiplet 101 are used to specify or indicate the data type used for the computation.

[0058] Each compute core can use 4B for each operand, i.e. four INT8 elements per operand, or two BFLOAT / HALF / INT16 elements per operand. An operand is an input read from the example scratchpad memory (described below) and broadcast across the compute cores. Figure 5 The broadcast function is described. The second operand is the weight value read from the SIMD register 310. In some implementations, the weight value for each core is different.

[0059] As shown in Table 1 below, the techniques and architectures described in this specification can be used to accelerate computations for different combinations of K-in-N sparsity and data types (dt). Table 1: Combinations of K-in-N sparsity and data type (dt)

[0060] Each core element can perform a multiplication and accumulation function. The accumulation function is performed on the partial sum generated from the multiplication operation. The accumulation function can be expressed as: (MAC(operand1(4B), operand4 (4B), partial_sum(32B)->partial_sum'(32B)). In some implementations, each computing unit or MAC includes a core element accumulator, and the partial sum is stored in the core element accumulator as a partial result.

[0061] exist Figure 3 and Figure 4In the example of , memory 108 can be implemented as scratchpad memory 308, 408, respectively. In some implementations, one or more memory structures at the compute chiplet 101 can be implemented as scratchpad memory, such as shared scratchpad memory. The memory structure can be configured to support a direct memory access (DMA) mode that atomically reduces incoming vector data to a memory location, for example, configured to support atomic floating point reductions rather than simply overwriting the data. In some implementations, the resources of shared memory 308, 408 are configured to be used as a software-controlled scratchpad rather than, for example, a hardware-managed cache.

[0062] exist Figure 3 and Figure 4 In each of , a Boolean "0" corresponds to a detected zero weight value, and a Boolean "1" corresponds to a detected non-zero weight value. Figure 3 In the example of , the mapping vector 302 is a 4-bit vector, and Figure 4 In the example of , the mapping vector 402 is an 8-bit vector. Each of the mapping vectors 302, 402 may also have more or fewer bits. For example, each of the mapping vectors 302, 402 may vary in size (or bits) based on design preferences, processing constraints, system configuration, or a combination of these.

[0063] In some implementations, one or more of the mapping vectors 302, 402 allow for simplified processing of neural network operations of a machine learning data set at the compute chiplet 101. In order to optimize the degree to which neural network operations can be simplified relative to conventional methods, the system 100 utilizes a specific hardware architecture of an application-specific integrated circuit that uses corresponding pairing of broadcast input buses coupled to a group of compute cores across multiple compute chiplets to accelerate the execution of artificial neural networks. Figure 5 This is explained in detail with the example of .

[0064] Figure 5 An example processing pipeline 500 is shown for routing inputs obtained from memory locations of memory 108 to one or more compute cores 114. Pipeline 500 utilizes a hardware architecture in which input bus 106 is coupled (e.g., directly coupled) to each of a plurality of hardware compute core groups of application specific integrated circuits. System 100 provides inputs or activations (e.g., a0, a1, a2, etc.) of input feature maps to a subset of MACs 114.

[0065] For example, the corresponding input of the input vector 102 is provided to each MAC in the subset via the input bus 106 of the compute chiplet 101. The system 100 can perform this broadcast operation across multiple compute chiplets 101 to calculate the product for a given neural network layer using the corresponding input grouping and corresponding weights at each compute chiplet 101. At a given compute chiplet 101, the product is calculated by multiplying the corresponding input (e.g., a1) with the corresponding weight (e.g., w1) at each MAC in the subset using the multiplication circuitry of the MAC.

[0066] The system 100 may generate an output of a layer based on the accumulation of multiple corresponding products computed at each MAC 114 in the subset. Figure 7 As explained, a multiplication operation performed within the compute chiplet 101 may involve: i) a first operand (e.g., input or activation) corresponding to a corresponding element of an input tensor stored at a memory location of the memory 108; and ii) a second operand (e.g., weight) corresponding to a corresponding element of a parameter tensor stored at a memory location of the memory 110.

[0067] exist Figure 5 In the example of , the shift register 502 may provide shift functionality in which inputs of operands 504 are broadcast onto the input bus 106 and routed to one or more MACs 114. In some implementations, the shift register 502 enables one or more input broadcast modes at the compute chiplet 101. For example, the shift register 502 may be used to broadcast inputs sequentially (one by one) from the memory 108, broadcast inputs simultaneously from the memory 108, or use some combination of these broadcast modes. The shift register 502 may be an integrated function of the memory 108 and may be implemented in hardware, software, or both.

[0068] As shown, in one implementation, the weight (w2) of operand 506 may have a zero weight value. When the controller 103 determines that the weight (w2) has a zero value, in order to save processing resources, the multiplication between the input (a2) and the weight (w2) may be skipped so that these operands are not routed to or consumed by the core 114 a / b / c. As described above, the determination to skip this particular multiplication operation may be based on a mapping vector that maps discrete inputs (an) of the input vector to individual weights (wn) of the compressed sparse parameters.

[0069] Figure 6 is a flow chart of an example process 600 for exploiting data sparsity during computation of a neural network implemented on a hardware accelerator. In some examples, computations are performed using a dedicated hardware integrated circuit to process a neural network input such as an image or a speech utterance.

[0070] For example, a hardware integrated circuit may be configured to implement a CNN including multiple neural network layers. In some cases, the neural network layers may include grouped convolutional layers. The input may be an example image including various other types of digital images or related graphical data as described above. In at least one example, the integrated circuit implements an RNN for processing input derived from speech or other audio content. In some implementations, process 600 is part of a technique for accelerating neural network computations, which also allows for improved accuracy of image or speech processing output relative to other data processing techniques.

[0071] Process 600 may be implemented or performed using system 100 described above. Thus, the description of process 600 may refer to the above-mentioned computing resources of system 100. In some examples, the steps or actions of process 600 are implemented by programmed firmware instructions, software instructions, or both; wherein each type of instruction may be executed by one or more processors of the devices and resources described in this document.

[0072] In some implementations, the steps of process 600 are performed at a hardware circuit to generate a layer output of a neural network layer. The output can be part of a computation that generates an image processing or image recognition output for a machine learning task or inference workload. The integrated circuit can be a dedicated neural network processor or hardware machine learning accelerator configured to accelerate computations for generating various types of data processing outputs.

[0073] Referring again to process 600, an input vector is obtained from a first memory of a compute chiplet 101 of the hardware accelerator (602). For example, the controller 103 of the compute chiplet 101 generates a control signal to obtain the input vector from an address location of the first memory 102. In some implementations, the input vector corresponds to an input feature map of an image and can be a matrix structure of a neural network input such as activations generated by a previous neural network layer.

[0074] The compute chiplet 101 processes an opcode indicating sparsity among data for neural network computations (604). For example, the controller 103 may process the opcode in response to scanning, analyzing, or otherwise reading one or more instructions received at the compute chiplet. The opcode may indicate the sparsity of an example parameter tensor including weights of one or more neural network layers. The parameter tensor and its corresponding weight (parameter) values ​​are stored in a memory of the hardware accelerator, such as the second memory 110, and may be accessed from the memory. For example, the parameter tensor may be an 8×8 matrix having one or more columns or rows having K-in-N sparsity, where K=2 and N=4 (i.e., 2-in-4 sparsity).

[0075] As described below, the controller 103 reads the opcode and configures the computation of the neural network layer at the compute chiplet 101 to exploit the sparsity indicated by the opcode. The opcode may be predetermined by a compiler of an integrated circuit (e.g., a hardware accelerator) that generates a set of instructions for execution at one or more compute chiplets of the circuit. For example, the compiler may be operable to generate a set of instructions for performing neural network computations for a machine learning workload such as an inference or training workload in response to compiling source code. The set of instructions may include a single instruction that is broadcast to one or more compute chiplets 101.

[0076] The corresponding single instruction consumed by a given compute chiplet may specify an opcode indicating the sparsity of the parameter tensor assigned to the compute chiplet. The opcode may also indicate an operation to be performed at the compute chiplet 101 to exploit the tensor sparsity. In some implementations, the opcode instructs the compute chiplet 101 to generate and execute localized instructions or control signals based on the compressed sparse parameters and corresponding mapping vectors received at the compute chiplet, generated locally at the compute chiplet 101, or both, to exploit the tensor sparsity.

[0077] In some implementations, the opcode is dynamically determined at runtime, for example, by a higher-level controller 125 of the hardware accelerator or a local controller 103 of a compute chiplet 101. The opcode can be a unique opcode included among a plurality of opcodes broadcast by the higher-level controller to a plurality of compute chiplets of the hardware accelerator. As described above, in some implementations, the plurality of opcodes are broadcast in a single instruction provided to each of the plurality of chiplets 101 of the hardware accelerator.

[0078] A compressed sparse parameter set is derived from a parameter tensor obtained based on an opcode (606). The opcode in a single instruction received at the compute chiplet 101 may indicate a 2-in-4 sparsity for a given column or row dimension of one or more parameter tensors. For example, the opcode may indicate that multiple columns 202 of the parameter tensor 200 have a pattern (e.g., a sparsity pattern) where for every four elements along a given dimension, two of the four elements are assigned zero-valued weights and two of the four elements are assigned non-zero-valued weights. The four elements may be {w0, w1, w2, w4}, where weights w1, w4 have non-zero weight values. Based on this, the controller 103 may detect these non-zero weight values ​​and then access (or generate) a compressed sparse parameter set, represented as {w1, w4}.

[0079] A mapping vector is generated based on the compressed sparse parameter set and the opcode (608). In some implementations, the mapping vector is generated external to the compute chiplet 101 and then provided to and stored at the compute chiplet 101. Based on a parameter tensor including four elements {w0, w1, w2, w4} and non-zero weight values ​​of weights w1, w4, the controller 125 can then generate a mapping vector with reference to an input vector for which the weights of the four elements {w0, w1, w2, w4} are to be processed. The mapping vector can be based on an encoding scheme that indicates the location of zero and non-zero weight values ​​of the corresponding parameter tensor. In some other implementations, the controller 103 can perform operations to generate a mapping vector locally at a given compute chiplet 101.

[0080] The opcode may include one or more fields in an instruction (e.g., a single instruction) received at the compute chiplet 101. In some implementations, the opcode may include respective values ​​of a first data item K and a second, different data item N. The compute chiplet 101 is operable to support various options for implementing sparse computations as indicated by at least the data values ​​of K and N. For example, the compute chiplet 101 enumerates the options based on the data value of the opcode in the instruction received at the compute chiplet 101 (including the values ​​of one or more fields of the opcode).

[0081] For example, an opcode with a data value of N=8 and specifying a data type of int8 (1B) (e.g., for integer data) informs the controller 103 that a particular sparse computation requires the activation of reading or retrieving N*1B from the memory 108. Similarly, the opcode may instruct the controller 103 to perform a parameter read from the SIMD register 310. For example, the controller 103 retrieves N bits from the non-zero mapping vector (e.g., NZM) 302 and retrieves K*1B elements from the CSP 305, and activates the appropriate selection logic to perform the sparse computation. Retrieving the required NZM bits and corresponding CSP elements of the parameters involves reading the corresponding address space in the SIMD register 310 that stores some (or all) of these data items.

[0082] You can refer to Figure 3 101 . From the perspective of "cell0" in compute cell 112 and considering arithmetic operations relative to uncompressed tensors, compute chiplet 101 needs to perform an example dot product: a0*w0+a1*w1+a2*w2+a3*w3, where {a0, a1, a2, a3} is an example activation vector and {w0, w1, w2, w3} is an example weight vector corresponding to the sparse parameter tensor. Compute chiplet 101 analyzes each corresponding value of the weights in the weight vector and generates a mapping vector based on these values.

[0083] For example, the compute chiplet 101 may access, retrieve, or otherwise generate a mapping vector corresponding to weights {w0, w1, w2, w3} having a bitmap of

[0101] , where each bit of the bitmap corresponds to a corresponding value of a weight in the weight vector. In this example, the bitmap

[0101] indicates that each of the weights w0 and w2 has a zero value. The compute chiplet 101 may also access, retrieve, or otherwise generate compressed sparse parameters based on each corresponding value of the weight in the weight vector. For example, the controller 125 may derive a compressed set of parameters {w1, w3} from the weights {w0, w1, w2, w3}, indicating that each of the weights w1 and w3 has a non-zero value. Alternatively or additionally, the compute chiplet 101 may also locally derive a compressed set of parameters based on operations performed by its local controller 103.

[0084] In some implementations, the compute chiplet 101 obtains or generates a compressed sparse parameter CSP0 and associates the compressed set of parameters {w1, w3} with the compressed sparse parameter CSP0. Each of the mapping vectors

[0101] and CSP0 may be stored and later accessed from a corresponding address location of a SIMD register 310 of the memory 110. For example, the example mapping vector {0101} is stored at the NZM address of the SIMD register 310, and the parameter CSP0 is stored at the CSP address of the SIMD register 310.

[0085] Using the mapping vector

[0101] and CSP0{w1, w3}, the compute chiplet 101 can perform a dot product calculation at cell0. For example, the compute chiplet 101 can perform the dot product calculation a1*w1 + a3*w3. In some implementations, to simplify the calculation, the compute chiplet 101 can automatically initialize the multiplier in cell0 based on the non-zero weight value associated with the parameter CSP0. The compute chiplet 101 can then extract the inputs a1 and a3 from the {a0, a1, a2, a3} activation vector based on the bitmap of the mapping vector. For example, the selection logic 314 of the compute chiplet 101 references the mapping vector to align its extraction of the inputs a1 and a3 with the corresponding non-zero weight values ​​{w1, w3} of CSP0.

[0086] The input vector is processed through the layers of the neural network using the mapping vector and the compressed sparse parameter set (610). In some implementations, the compute chiplet 101 performs a dot product matrix multiplication operation to process the input vector through the layers of the neural network. The compute chiplet 101 performs a dot product matrix multiplication operation between the inputs of the input vector and the corresponding weight values ​​in the compressed sparse parameter set.

[0087] For example, the operation includes multiplying the input or activation value with the weight value over one or more cycles to generate multiple products (e.g., partial sums), and then performing accumulation of these products over multiple cycles. For each input, a dot product matrix multiplication operation can be performed at one or more multiplication cores of the hardware accelerator based on the corresponding bit value of each bit in the mapped vector.

[0088] In some implementations, the compute chiplet 101 performs convolution operations alone or in collaboration with other compute chiplets to process an input vector through a layer of a neural network. For example, the system 100 can perform convolution at a neural network layer by processing an input feature map (e.g., an input vector) using a compute core of a hardware integrated circuit. The core can be a hardware multiply-accumulate core of a hardware compute unit at the hardware integrated circuit.

[0089] Additionally, processing the input also includes: providing the weights of the compressed sparse parameters to a subset of the multiply-accumulate core elements of the hardware accelerator. In some implementations, the controller 103 determines a mapping of the input of the input vector to the core elements of the compute chiplet based on one or more opcodes in an instruction (e.g., a single instruction) received at the compute chiplet 101. The controller 103 generates a control signal 105 based on the determined mapping to provide the weights of the compressed sparse parameters to the subset of the multiply-accumulate core elements. As described above, the weights of the compressed sparse parameters are provided from an example SIMD register.

[0090] When performing neural network calculations for N=4, K=2, dt=half / bfloat / int16 (2 bytes), the compute chiplet 101 can be configured to: i) read four bits of the mapping vector (e.g., {0101}) stored at the NZM address and two compressed parameters (4B) (e.g., {w1, w3}) stored at the CSP address; ii) read four elements (8B) of the activation operand from the temporary storage 308; iii) use the four bits of the mapping vector (e.g., {0101}) to select the correct two compressed parameters (4B) (e.g., {w1, w3}) as the corresponding weight operands; iv) feed the appropriate activation operands and corresponding weight operands to the compute core to perform multiplication and accumulation operations; and v) increment the relevant addresses in the memory to read the next activation operand and weight operand from the memory.

[0091] When performing neural network calculations for N=8, K=4, dt=int, the compute chiplet 101 can be configured to: i) read the eight bits of the mapping vector stored at the NZM address (e.g., {01011001}) and the four compressed parameters (4B) stored at the CSP address (e.g., {w1, w3, w11, w7}); ii) read the eight elements (8B) of the activation operand from the temporary storage 408; iii) use the eight bits of the mapping vector (e.g., {01011001}) to select the correct four compressed parameters (4B) (e.g., {w1, w3, w11, w7}) as the corresponding weight operands; iv) feed the appropriate four operands (activation and corresponding weights) to the compute core to perform a multiplication-accumulation operation; and v) increment the relevant address in the memory to read the next activation and weight operands from the memory.

[0092] When performing neural network calculations for N=8, K=2, dt=half / bfloat / int16, the compute chiplet 101 can be configured to: i) read eight bits of the mapping vector stored at the NZM address (e.g., {01000001}) and two compressed parameters (4B) stored at the CSP address (e.g., {w1, w3,}); ii) read eight elements (16B) of the activation operand from the temporary storage 408; iii) use the eight bits of the mapping vector (e.g., {01000001}) to select the correct two compressed parameters (4B) (e.g., {w1, w3}) as the corresponding weight operands; iv) feed the appropriate two operands (activation and corresponding weight) to the compute core to perform a multiplication-accumulation operation; and v) increment the relevant address in the memory to read the next activation and weight operands from the memory.

[0093] When performing neural network calculations for N=16, K=4, dt=int, the compute chiplet 101 can be configured to: i) read 16 bits of the mapping vector stored at the NZM address (e.g., {01000001 00001100}) and four compressed parameters (4B) stored at the CSP address (e.g., {w1, w3}); ii) read 16 elements (32B) of the activation operand from the temporary storage 408; iii) use the 16 bits of the mapping vector (e.g., {01000001 00001100}) to select the correct four compressed parameters (4B) (e.g., {w1, w3, w11, w7}) as the corresponding weight operands; iv) feed the appropriate four operands (activation and corresponding weights) to the compute core to perform a multiplication-accumulation operation; and v) increment the relevant address in the memory to read the next activation and weight operands from the memory.

[0094] When performing neural network calculations for N=16, K=2, dt=half / bfloat / int16, the compute chiplet 101 can be configured to: i) read 16 bits of the mapping vector stored at the NZM address (e.g., {01000000 00000100}) and two compressed parameters (4B) stored at the CSP address (e.g., {w1, w3,}); ii) read 16 elements (32B) of the activation operand from the temporary storage 408; iii) use the 16 bits of the mapping vector (e.g., {0100000100001100}) to select the correct two compressed parameters (4B) (e.g., {w1, w3}) as the corresponding weight operands; iv) feed the appropriate two operands (activation and corresponding weight) to the compute core to perform a multiply-accumulate operation; and v) increment the relevant address in the memory to read the next activation and weight operands from the memory.

[0095] When performing neural network calculations for N=32, K=4, dt=int, the compute chiplet 101 can be configured to: i) read the 32 bits of the mapping vector stored at the NZM address and the four compressed parameters (4B) stored at the CSP address; ii) read the 32 elements (32B) of the activation operand from the temporary storage 408; iii) use the 32 bits of the mapping vector to select the correct four compressed parameters (4B) as the corresponding weight operands; iv) feed the appropriate four operands (activation and corresponding weights) to the compute core to perform a multiplication-accumulation operation; and v) increment the relevant address in the memory to read the next activation and weight operands from the memory.

[0096] Figure 7 An example of a tensor or multi-dimensional matrix 700 including an input tensor 704, a variation of a parameter tensor 706, and an output tensor 708 is illustrated. Figure 7 In the example of , each of tensors 700 includes corresponding elements, where each element can correspond to a corresponding data value (or operand) of a computation performed at a given layer of the neural network.

[0097] For example, each input of the input tensor 704 may correspond to a corresponding element along a given dimension of the input tensor 704, each weight of the parameter tensor 706 may correspond to a corresponding element along a given dimension of the parameter tensor 706, and each output value or activation in the output set may correspond to a corresponding element along a given dimension of the output tensor 708. Relatedly, each element may correspond to a corresponding memory location or address in the memory of the compute chiplet 101 assigned to operate on one or more dimensions of the given tensor 704, 706, 708.

[0098] The computations performed at a given neural network layer may include multiplying input / activation tensor 704 with parameter / weight tensor 706 over one or more processor clock cycles to produce a layer output, which may include an output activation. Multiplying activation tensor 704 with weight tensor 706 includes multiplying activations from elements of tensor 704 with weights from elements of tensor 706 to produce one or more partial sums. Figure 7 The example tensors 706 of can be unmodified parameter tensors, modified parameter tensors, or a combination thereof. In some implementations, each parameter tensor 706 corresponds to a modified parameter tensor that includes a non-zero CSP value derived based on the sparsity exploitation techniques described above.

[0099] The processor core of the system 100 can operate on: i) a scalar corresponding to a discrete element in a multidimensional tensor 704, 706; ii) a vector (e.g., input vector 102) comprising values ​​of multiple discrete elements 707 along the same or different dimensions of a multidimensional tensor 704, 706; or iii) a combination of these. Depending on the dimensionality of the tensor, a discrete element 707 in a multidimensional tensor or each discrete element in multiple discrete elements 707 can be represented using X, Y coordinates (2D) or using X, Y, Z coordinates (3D).

[0100] The system 100 can calculate multiple partial sums corresponding to the products generated by multiplying a batch of inputs with corresponding weight values. As described above, the system 100 can perform accumulation of products (e.g., partial sums) over multiple clock cycles. For example, the accumulation of products can be performed in a random access memory, a shared memory, or a temporary storage of one or more computing chiplets based on the techniques described in this document. In some implementations, the input-weight multiplication can be written as a sum of products of each weight element multiplied by a discrete input (such as a row or slice of the input tensor 704) of the input vector 102. The row or slice can represent a given dimension, such as a first dimension 710 of the input tensor 704 or a second different dimension 715 of the input tensor 704.

[0101] In some implementations, the output of a convolutional neural network layer may be calculated using an example set of calculations. The calculations for the CNN layer may involve performing a 2D spatial convolution between a 3D input tensor 704 and at least one 3D filter (weight tensor 706). For example, convolving one 3D filter 706 onto the 3D input tensor 704 may produce a 2D spatial plane 720 or 725. The calculations may involve calculating the sum of dot products for a particular dimension of the input volume including the input vector 102.

[0102] For example, spatial plane 720 may include output values ​​of sums of products computed from inputs along dimension 710, while spatial plane 725 may include output values ​​of sums of products computed from inputs along dimension 715. The computation of sums of products for generating output values ​​in each of spatial planes 720 and 725 may be performed i) at compute cores 114 a / b / c, ii) directly at memory 110 using arithmetic operators coupled to a shared memory bank of memory 110, iii) or both. In some implementations, the reduction operation may be simplified and performed directly at the memory cores (or locations) of memory 110 using various techniques for reduction of accumulated values.

[0103] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or to control the operation of a data processing device.

[0104] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode the information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0105] The term "computing system" encompasses all kinds of equipment, devices and machines for processing data, including by way of example a programmable processor, a computer or multiple processors or computers. The equipment may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0106] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0107] A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store one or more modules, subroutines, or code portions. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0108] The processes and logic flows described in this specification can be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0109] Computers suitable for executing computer programs include, by way of example, central processing units that can be based on general or special microprocessors or both, or any other kind of microprocessor. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. Some elements of a computer are a central processing unit for fulfilling or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or operatively coupled to receive data from the one or more mass storage devices or to transfer data to the one or more mass storage devices, or both. However, a computer need not have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0110] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including by way of example semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0111] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0112] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0113] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0114] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a change in a sub-combination.

[0115] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed, to achieve the desired result. In some contexts, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0116] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method involving a neural network implemented on a hardware accelerator, the method include: derive a compressed sparse parameter set from the parameter tensor; generating a mapping vector based on the compressed sparse parameter set; processing instructions that instruct to perform a sparse computation using the compressed sparse parameters based on the sparsity of the parameter tensor; obtaining, based on the instruction, i) an input vector from a first memory of the hardware accelerator and ii) the compressed sparse parameters from a second memory of the hardware accelerator; as well as The sparse computation is performed using the mapping vector and the compressed set of sparse parameters to process the input vector through a layer of the neural network.

2. The method according to claim 1, in, Processing the input vector through the layers of the neural network includes: A dot product matrix multiplication operation is performed between the input of the input vector and the corresponding weight values ​​in the compressed sparse parameter set.

3. The method according to claim 2, in, The dot product matrix multiplication operation is performed at one or more multiplication cores of the hardware accelerator based on the corresponding bit value of each bit in the mapped vector.

4. The method of claim 2, further comprising: include: accessing hardware selection logic of the first memory and the second memory coupled to the hardware accelerator; as well as Using the hardware selection logic, a particular input of the input vector and a corresponding weight value in the compressed sparse parameter set are selected based on respective bit values ​​of bits in the mapping vector.

5. The method according to claim 1, in, Deriving the compressed sparse parameter set includes: Generates a modified parameter tensor that includes only non-zero elements along a particular dimension of the parameter tensor.

6. The method according to claim 5, in, Generating the modified parameter tensor includes: For a specific column dimension of the parameter tensor: generating a compressed representation of the column dimension based on the non-zero elements of the column dimension; and Concatenate each non-zero element in the compressed representation of the column dimension.

7. The method according to claim 6, in, Generating the modified parameter tensor includes: The corresponding dimension position of each non-zero element in the parameter tensor is preserved before generating the modified parameter tensor.

8. The method according to claim 1, in: The parameter tensor comprises a plurality of dimensions; and An opcode in the instruction indicates sparsity of a particular dimension of the plurality of dimensions.

9. The method according to claim 1, in: The hardware accelerator is operable to process a plurality of multi-dimensional parameter tensors; and An opcode in the instruction indicates uniform sparsity across each of the plurality of multi-dimensional parameter tensors.

10. The method according to claim 1, in, The first memory: is a temporary storage device of the hardware accelerator; and Configured to store inputs and activations processed at the neural network layer.

11. The method according to claim 10, in, The second memory comprises a plurality of single instruction multiple data (SIMD) registers, and the method comprises: storing the mapping vector at a first address of a SIMD register; and The compressed sparse parameter set is stored at a second different address of the SIMD register.

12. A system, include: a hardware accelerator configured to implement a neural network; Processing device; as well as A non-transitory machine-readable storage medium storing instructions executable by the processing device to cause operations including: derive a compressed sparse parameter set from the parameter tensor; generating a mapping vector based on the compressed sparse parameter set; processing instructions that instruct to perform a sparse computation using the compressed sparse parameters based on the sparsity of the parameter tensor; obtaining, based on the instruction, i) an input vector from a first memory of the hardware accelerator and ii) the compressed sparse parameters from a second memory of the hardware accelerator; as well as The sparse computation is performed using the mapping vector and the compressed set of sparse parameters to process the input vector through a layer of the neural network.

13. The system of claim 12, in, Processing the input vector through the layers of the neural network includes: A dot product matrix multiplication operation is performed between the input of the input vector and the corresponding weight values ​​in the compressed sparse parameter set.

14. The system of claim 13, in, The dot product matrix multiplication operation is performed at one or more multiplication cores of the hardware accelerator based on the corresponding bit value of each bit in the mapped vector.

15. The system of claim 13, in, The operations further include: accessing hardware selection logic for the first memory and the second memory coupled to the hardware accelerator; and Using the hardware selection logic, a particular input of the input vector and a corresponding weight value in the compressed sparse parameter set are selected based on respective bit values ​​of bits in the mapping vector.

16. The system of claim 12, in, Deriving the compressed sparse parameter set includes: Generates a modified parameter tensor that includes only non-zero elements along a particular dimension of the parameter tensor.

17. The system of claim 16, in, Generating the modified parameter tensor includes: For a specific column dimension of the parameter tensor: generating a compressed representation of the column dimension based on the non-zero elements of the column dimension; and Concatenate each non-zero element in the compressed representation of the column dimension.

18. The system of claim 17, in, Generating the modified parameter tensor includes: The corresponding dimension position of each non-zero element in the parameter tensor is preserved before generating the modified parameter tensor.

19. The system of claim 12, in: The parameter tensor comprises a plurality of dimensions; and An opcode in the instruction indicates sparsity of a particular dimension of the plurality of dimensions.

20. The system of claim 12, in: The hardware accelerator is operable to process a plurality of multi-dimensional parameter tensors; and An opcode in the instruction indicates uniform sparsity across each of the plurality of multi-dimensional parameter tensors.

21. The system of claim 12, in, The first memory: is a temporary storage device of the hardware accelerator; and Configured to store inputs and activations processed at the neural network layer.

22. The system of claim 21, in, The second memory includes a plurality of single instruction multiple data (SIMD) registers, and the operations further include: storing the mapping vector at a first address of a SIMD register; and The compressed sparse parameter set is stored at a second different address of the SIMD register.

23. A non-transitory machine-readable storage medium storing instructions executable by a processing device configured as a hardware accelerator for implementing a neural network, in, Execution of the instructions causes the execution of the following operations: derive a compressed sparse parameter set from the parameter tensor; generating a mapping vector based on the compressed sparse parameter set; processing instructions that instruct to perform a sparse computation using the compressed sparse parameters based on the sparsity of the parameter tensor; obtaining, based on the instruction, i) an input vector from a first memory of the hardware accelerator and ii) the compressed sparse parameters from a second memory of the hardware accelerator; as well as The sparse computation is performed using the mapping vector and the compressed set of sparse parameters to process the input vector through a layer of the neural network.

24. A method performed using a hardware accelerator, the hardware accelerator implementing a neural network comprising a plurality of neural network layers, the method include: Receiving instructions for a compute chiplet of the hardware accelerator, the instructions executable at the compute chiplet to cause execution of operations including: identifying an opcode in the instruction indicating sparsity of the parameter tensor; loading a compressed sparse parameter set based on weight values ​​derived from a parameter tensor specifying a plurality of weights of a layer of the neural network; Loading a mapping vector generated based on the compressed sparse parameter set; obtaining, based on the opcode, i) an input vector from a first memory of the hardware accelerator and ii) the compressed sparse parameter set from a second memory of the hardware accelerator; as well as Based on the mapping vector, the input vector is processed through the layer of the neural network using the compressed sparse parameter set.

Images