Area-efficient asymmetric multiply-and-accumulate unit design for various hybrid quantization support

Abstract

Methods and systems are described for performing multiply accumulate operations on input data with multiple bit widths. A multiply accumulate device that includes a first multiply unit having a first bit width and a second multiply unit having a second bit width that is smaller than the first bit width. The device is reconfigurable to perform a first multiplication operation using the first multiply unit when input data comprises input operands having the first bit width and a plurality of second multiplication operations using the first multiply unit and the second multiply unit when input data comprises input operands having the second bit width.

Description

AREA-EFFICIENT ASYMMETRIC MULTIPLY- AND-ACCUMULATE UNIT DESIGN FOR VARIOUS HYBRID QUANTIZATION SUPPORTBACKGROUND

[0001] This specification generally relates to an architecture for executing multiply-and- accumulate (MAC) operations in a computational device.

[0002] Multiply-and-accumulate (“MAC”) operations are important to the functionality of neural networks, serving as a key computational task for many neural network models. These operations multiply neural input values (activations) by their corresponding weights (parameters) and then accumulate these products over successive layers.

[0003] Modem computing systems include various processing devices that each offer different computing capabilities and trade-offs for executing neural network workloads. At least one device can be a processor-in-memory (“PIM”) that includes processing capabilities in a memory device.SUMMARY

[0004] This specification describes techniques for supporting one or more multiply-and- accumulate (“MAC”) units that perform MAC operations on multiple data types. These techniques utilize a computing architecture that supports MAC operations between operands of multiple different data types, while minimizing the number of hardware components required by re-using hardware components for calculations involving the multiple datatypes.

[0005] MAC operations are foundational to neural network calculations, playing a pivotal role in the efficiency and speed of these complex computational processes. In neural networks, MAC operations are used extensively during the forward pass to compute the weighted sum of inputs and their corresponding weights. Each MAC operation involves multiplying two operands (an input value and a weight) and then accumulating (adding) the result into a running sum. This process is repeated across all neurons and layers, making MACs crucial for the execution of tasks such as image recognition, natural language processing, and other artificial intelligence driven applications. The performance of a neural network, in terms of speed and energy efficiency, is often directly correlated with the ability of the hardware to execute MAC operations rapidly and effectively.

[0006] Devices can be designed to efficiently perform MAC operations, essential for executing neural network calculations. Traditional processors, such as central processing units (“CPU’7) and graphical processing units (“GPU”), are commonly used for this purpose.with GPUs being used for their parallel processing capabilities that accelerate the computation of numerous MAC operations simultaneously. Specialized hardware accelerators like Tensor Processing Units (“TPU”) and Field-Programmable Gate Arrays (“FPGA”) are optimized for high-throughput MAC operations, significantly enhancing the performance of machine learning tasks. Additionally, PIM devices represent an approach that integrates memory and processing power to perform MAC operations directly within memory chips, reducing data movement and improving efficiency. These devices underscore the hardware landscape developed to support the computational requirements of neural networks and artificial intelligence applications.

[0007] Area efficiency and energy efficiency are two optimization parameters in circuit design. Some circuit architectures focus on optimizing energy’ efficiency, while others focus on optimizing area efficiency. Area efficiency involves the optimization of circuit functionality to occupy the smallest possible physical space on a semiconductor wafer. It is a measure of how effectively the silicon area is utilized to perform the necessary electronic functions. High area efficiency means that a chip can provide more functionality, performance, or storage capacity within a given silicon area, leading to smaller, more compact, and often more cost-effective devices. Area efficiency is an important factor in circuit design for mobile and embedded devices that implement technology' like PIM.

[0008] As neural network models grow, e.g., the number of parameters (weights) increases, new techniques are increasingly researched for compressing models, such as those used in natural language processing (“NLP”), to reduce their file size while maintaining performance. This trend involves optimizing the bit width (e.g., 8-bit, 16-bit, etc.) of the model's data, particularly the weights, to use less memory. Techniques like quantization, where the precision of the weights is reduced from, for example, 32-bit floating points to 8- bit integers, are commonly employed. This not only shrinks the model size significantly, but in some cases, can speed up computation and / or reduce energy consumption. Compression strategies like quantization are important for deploying advanced models on devices with limited resources, such as mobile phones and embedded systems.

[0009] Designing systems that can accommodate MAC operations between operands with various bit widths is increasingly important in the face of innovation in compression, the increasing size of neural network models, and the uncertain trajectory of the data types of weights and activations. Arithmetic logic units (“ALU”) are the computational units that perform MAC operations. An ALU is designed to accommodate a particular bit width of the operands, e.g., the weight (parameter of the model) and the activation. For example, aparticular ALU that is designed to accommodate a multiplication of two 8-bit integers cannot accommodate a multiplication of two 16-bit integers. To address the aforementioned challenge, this specification describes techniques for designing a system to support multiple quantization schemes that includes a multiplication of weights and activations of various bit widths while minimizing the number of ALUs required to support the full range of possible bit width combinations.

[0010] In general, the technique described in this specification can be applied to any computing platform. However, it is particularly useful for machine learning ("MI.") accelerators that place a premium on area-efficient circuit design such as PIM devices.

[0011] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The techniques described in this specification provide an area-efficient implementation of a system of multiple ALUs that perform MAC operations for multiple combinations of bit widths (data types) betw een w eights of a neural network model and input activations. This provides a flexible platform to perform MAC operations for a variety of data types without requiring a specialized circuit for each operation type. By re-using ALUs and other hardware resources for multiple operation types, the number of circuit elements can be minimized while supporting a maximum number of weight-activation multiplication operations.

[0012] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardw are, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0013] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1 is a block diagram of an example computing system that uses a PIM device.

[0015] Fig. 2 illustrates an example arithmetic logic unit.

[0016] Fig. 3 is a block diagram of an example computing system that implements hybrid quantization logic.

[0017] Fig. 4 is a block diagram of an example computing system that implements hybrid quantization logic and area-efficient hardware quantization.

[0018] Fig. 5 is a block diagram of an example computing system that implements areaefficient hardware quantization.

[0019] Fig. 6 illustrates an example of zero-point subtraction logic.

[0020] Fig. 7 illustrates an example process that implements area-efficient hardware quantization.

[0021] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0022] This specification describes a system and techniques for supporting one or more multiply-and-accumulate (“MAC”) units that perform MAC operations on multiple data types. In particular, the system includes hardware components that are shared between multiple data operations for minimizing the number of required hardware components, and thus reducing an amount of required on-chip area.

[0023] FIG. 1 is a block diagram of an example computing sy stem 100 that uses a memory device 122. The memory device 122 is has a processor-in-memory (“PIM”) architecture, and is an example system that can execute MAC operations and benefit from an area-efficient architecture. The area-efficient MAC operations can alternatively or in addition be installed in any other appropriate computing device. This specification describes one or more implementations that include one or more arrangements of arithmetic logic units (“ALU”) to perform multiply-and-accumulate (“MAC”) operations. In some cases, the implementations use an area-efficient architecture that includes re-using one or more hardware resources to support operations between operands with multiple data ty pes. Although this specification describes the one or more implementations in the context of specific compute architectures (e.g.. PIM), the implementations can be extended to any compute resource that performs MAC operations (e.g., servers, edge devices, etc ). The description of the computing system 100 is provided to lay a foundational basis for an environment that can benefit from the implementations described in the following figures.

[0024] In this example, the system 100 includes a host device, which is referred to as a system-on-a-chip (“SOC”) 102 and a memory device 122 that has a PIM architecture. In other words, the memory device 122 can serve as a DRAM device for the system 100 and can also receive and perform PIM operations on data stored in the DRAM.

[0025] In general, the SOC 102 can issue PIM requests to the memory7device 122 using a primary7PIM interface 140 and can receive error data back from the memory device 122 using a separate error interface 142.

[0026] The system 100 is an example of a subsy stem that can be installed on any7appropriate user device. In the example of FIG. 1, the system 100 is an integrated subsystem of an example user device 130, which can be a consumer electronic device or mobile device, which can be, for example, a smartphone 130a, a tablet computer 130b. a laptop 130c, a smartwatch 130d, another type of wearable device, an eNotebook, a Netbook, a smart speaker, or a mobile computer, to name just a few examples. In some other implementations, the system 100 is an integrated subsystem of a desktop computer, a network server, or any other appropriate cloud-based computing system.

[0027] In some implementations, computing systems like the one described in relation to the system 100 that operate on user devices 130 like the smartphone 130a and the smartwatch 130d optimize for area efficiency as well as power efficiency. An ability7to perform required computations using compute resources (e.g., a CPU) in the smallest area possible is an important aspect for the design of systems that operate on edge devices. In particular, the implementations described in relation to FIGS 2-6 relate to an architectural design that is optimized for area efficiency.

[0028] The SOC 102 includes a central processing unit 104 (“CPU 104”), a memory controller 105, a shared memory 106 (“memory 106”), a PIM resource manager 108. and a circuit block 110. In some implementations, system 100 can include multiple SOCs and any descriptions for the SOC 102 will apply equally7to each of the multiple SOCs that may7be included at system 100.

[0029] The CPU 104 can be a general-purpose CPU (e.g.. a single or multi-core CPU) that can execute the primary functionality of the user device 130. The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory7and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more applicationvalues, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0030] The shared memory 106 is a memory subsystem that can be shared by other components of the SOC 102. In the example of Fig. 1, the shared memory 106 is depicted as being external to the circuit block 110. However, the shared memory 106 can include portions of memory that are: i) specific to the circuit block 110. ii) external to the circuit block 110, or iii) both. The shared memory 106 can be random access memory of the SOC 102, such as static random access memory' (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0031] In some implementations, aspects of the shared memory 106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more components of the SOC 102. The shared memory' 106 can also include various other ty pes of memory subsystems, such as high bandwidth memory' (HBM), narrow memory (e.g., for storing 8-bit values), or wide memory (e.g., for storing 16-bit or 32-bit values), to name just a few examples.

[0032] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 can include one or more proprietary hardware elements. For example, each of the ISP 1 12, TPU 114, DSP 116, and GPU 118 can be a respective proprietary' IP block (or IP device) of a particular entity' or device manufacturer.

[0033] The PIM resource manager 108 can be implemented in hardware, software, or a combination of these. Aspects of the PIM resource manager 108 can be also implemented as firmware of the SOC 102 or firmware of a device of the SOC 102. For example, the PIM resource manager 108 can include resources such as flip-flops, registers, buffers, etc., that are implemented in hardware and can have control logic (e.g., programmed code) that is implemented in software. One or more aspects of the PIM resource manager 108 can be implemented as a software routine (or module) of the CPU 104, which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc.

[0034] The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from the shared memory 106. the memory device 122, or both. In some implementations, each processor (e.g., ISP 1 12, DSP 116, TPU 114, GPU 118)of the SOC 102 uses the PIM resource manager 108 to generate control signaling to manage and distribute PIM requests to perform memory -intensive compute operations to the memory device 122 to minimize the processing load at each core of the processors. The control signaling can be routed through the system 100 using an example bus 120 of the SOC 102. The control signaling can include commands, requests, data, instructions, or combination of these.

[0035] The PIM resource manager 108 cooperates with the CPU 104, memory controller 105 and storage controller 107 to dynamically control and manage one or more PIM operations to be performed by the memory' device 122.

[0036] The memory device 122 can include multiple memory dies, each having one or more memory arrays for implementing the memory functionality of the memory device 122. For example, the memory device 122 can include N memory dies, where N is an integer greater than 1. The memory device 122 can implement a dynamic random-access memory (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory' device 122 is configured to perform or support various types of PIM operations, which in this specification can also include compute-in-memory operations (CIM operations), and memory -near-computing operations (“MnC operations"). The memory' device 122 performs or supports these operations using its multiple PIM compute elements.

[0037] The SOC 102 cooperates with the memory device 122 to perform PIM computations across one or more memory dies 123 of the memory device 122. The computations can be for operations or workloads that involve processes executed by one or more of the processors in the circuit block 110. Alternatively or in addition, the PIM computations performed by the memory device can be part of a heterogeneous operation that spans multiple processors of circuit block 110, multiple circuit blocks 110, or both. In at least one example the memory device 122 is external to the SOC 102, whereas in another example the memory' device 122 can be internal to the SOC 102.

[0038] In basic operation, a component of the SOC 102 initiates a request to perform PIM operations on the memory device 122. For example, the component can communicate with the PIM resource manager 108 to provide a sequence of one or more instructions to be performed over a range of memory addresses by compute elements 126 of the memory' device 122. The SOC 102 then provides data representing the sequence of instructions and the range of memory addresses to the memory device 122 over the primary PIM interface 140.

[0039] The memory device 122 receives the data representing the sequence of instructions as well as the addresses over which to perform the PIM operations on theprimary PIM interface 140. The memory device 122 then distributes the instructions among the compute elements 126. Each of the compute elements 126 can be a separate processor that can perform instructions according to a PIM instruction set that defines supported PIM operations. In some implementations, the memory device 122 distributes the sequence instructions among the multiple compute elements 126 in a SIMD fashion.

[0040] After the PIM operations have completed, the memory device 122 can provide an indication that the operations have completed back to the SOC 102 over the primary PIM interface 140. The PIM resource manager 108 can then notify the requesting processing component that the PIM operations have completed and that the corresponding process can continue.

[0041] If an error occurs while performing the PIM operations, the memory device 122 can provide an indication of the error over the separate error interface 142. The PIM resource manager 108 can receive the error data over the error interface 142 and can take an appropriate remedial action. In some implementations, the PIM resource manager 108 can reset the state of the memory device 122 using the error data received over the error interface 142. The PIM resource manager 108 can then restart the PIM operations from an appropriate starting place, e g., from an appropriate instruction sequence, which can often avoid performing redundant computations in the face of an error.

[0042] In some implementations, the memory device 122 stores weights (parameters of a neural network model) that correspond to a particular neural network in one or more memory arrays 124. In some cases, the particular neural network is designed to perform many multiply -and-accumulate (“MAC”) operations for each layer of the neural network. The output activations of each layer of the neural network are fed to a subsequent layer of the neural network, where the output activations participate in a series of MAC operations with the weights of the corresponding subsequent layer. In some cases, each layer of the neural network is executed on a separate PIM device, where the corresponding weights are stored locally in the corresponding memory' device 122 in the one or more memory7arrays 124. In some other cases, weights of multiple layers are stored on a particular PIM device or are sequentially loaded onto the particular PIM device.

[0043] Fig. 2 illustrates an arithmetic logic unit (“ALU”) 200 that supports an execution of a multiply-and-accumulate (“MAC”) operation. The ALU 200 is designed to execute a multiplication task and an addition task in a single, continuous process.

[0044] The ALU 200 includes a multiplier 210 and an accumulator 220. For a single MAC operation, the multiplier 210 receives two input operands, e.g., a first operand 202associated with a set of weights of a machine learning model like a neural network model (neural network weights 230) and a second operand 204 associated with a set of activations of the neural network, e.g., a value of a neuron associated with a parameter that corresponds to a layer of the neural network (neural network activations 240). The set of neural network activations 240 are generated as the neural network computes the output of each layer of the neural network. The multiplier 210 multiplies the first operand 202 and the second operand 204 to generate an intermediate product 208. The accumulator 220 adds (212) the intermediate product 208 to a running sum of intermediate products. In some implementations, the accumulator 220 is a register that maintains a sum of all previous products computed during a sequence of MAC operations that corresponds to an evaluation of a particular output set of activations. The MAC operation is often implemented in an evaluation of a weighted sum, in which the value of the weighted sum is a sum of terms multiplied together. For example, a*x + b*y + c*z is a weighted sum where (x, y, z) is a first set of operands (e.g., input activations) and (a, b, c) is a second set of operands (e.g., model parameters or weights).

[0045] Each of the two input operands are represented by a particular data type. In addition, the ALU 200 is configured to operate on input operands with a range of particular data types. For example, in some implementations, the neural network parameters 230 are stored in a memory device (e.g., DRAM, SRAM, etc.).|0046| The memory device can store the neural network weights 230 with a particular bit width. For example, the memory device can store the neural network weights 230 as 8-bit integers, 16-bit integers, etc. As the bit width increases, the amount of memory' to store the neural netw ork w eights 230 increases. If a first set of neural network weights can be learned and stored with 8-bit integers and provide the same performance as a second set of neural network weights that are stored with 16-bit integers, the first set of neural network weights requires less memory' than the second set of neural network weights.

[0047] To accommodate multiple applications that operate on operands with multiple data types, it is advantageous for an ALU, or a set of ALUs, to be able to support MAC operations for a full range of data type combinations between the first operand 202 and the second operand 204 while optimizing for area-efficiency for PIM and other area-constrained devices.

[0048] Fig. 3 is a block diagram of an example computing system that implements hybrid quantization logic. Hybrid quantization logic supports MAC operations that include input operands of different data types (e.g., different bit widths) to be performed by an ALU with aparticular data type support. The data type support of the ALU is determined by the particular design of the ALU. For example, a particular ALU may only support 8-bit input operands. As another example, a particular ALU may only support 2-bit input operands. In some implementations, as the bit width of the supported data type of an ALU increases, the power and area requirements increase as well.

[0049] Quantization refers to a process of reducing the precision of numbers (e.g., weights of a neural network model). In some cases, the numbers are an operand of a MAC operation. Typically, neural network models are trained using floating-point numbers, which provide a high degree of precision at the cost of low computational and energy efficiency. As an example, a quantization process converts the floating-point numbers into lower-bit representations (e.g.. integers). In some cases, the quantization process does not meaningfully affect the trained neural network model’s performance, and the lower-bit representation leads to reduced memory' requirements and in some cases, greater power, and computational efficiency.

[0050] In some implementations, the hybrid quantization logic can enable an ALU to support a MAC operation between two operands that may not satisfy the ALU data type requirement. For example, the hybrid quantization logic can convert an input operand with a data type (e.g., bit width) of Fixedlntl (1-bit) to a data type of Fixedlnt8 to satisfy a data type requirement for a particular ALU that supports a MAC operation with Fixedlnt8 input operands. In some implementations, the hybrid quantization logic performs the data type conversion on the fly (e g., after the operand is retrieved from memory and before it is processed by the multiplier of the ALU, as described in relation to Fig. 2).

[0051] In some implementations, a DRAM bank 302 stores a set of w eights (e.g., model parameters) of a neural network model (e.g., large language model, etc.). In some cases, the DRAM bank 302 is part of a PIM (e.g., the memory device 122) that is performing the MAC operations associated with the hybrid quantization logic. The w eights are stored in the DRAM bank 302 with a particular data type, e.g., Fixedlnt8, Fixedlnt2, etc. In some cases, neural network models are designed such that the associated parameters are represented with a particular data type in a memory device (e.g., the DRAM bank 302). Data types with a smaller bit width consume less memory, and potentially less pow er and computing resources, than data types with large bit widths. A model that can be stored using w eights of a Fixedlnt2 data type is more memory efficient than a model that is stored using weights of a Fixedlnt8 data type. If a model can be demonstrated to perform with similar performance metrics, it can be advantageous to store the weights of the model in the data type that requires less memory.The DRAM bank 302 includes multiple sections, pages, columns, or rows that include a fixed amount of memory. For example, a memory section 304 of the DRAM bank 302 can store 32 bytes (32B) of data (e.g., 32B of data associated with the weights of the machine learning model). In some other implementations, the DRAM bank 302 can be an SRAM, or any other appropriate memory device.

[0052] An input cache 306 caches memory requests sent to the DRAM 302. Each memory request returns one or more associated weights from the DRAM 302 to a requester. In some implementations, the requester is a memory controller, an ALU, a CPU, or any intermediate or computational device that is involved in the processing of the weights of the corresponding neural network model. In this example, the cached data represents 32B of data, which corresponds to 256 bits (1 byte (B) = 8 bits).

[0053] The data type of the data of the input cache 306 is determined by the particular configuration of the neural network model. In other words, the weights of the neural network model (e.g., the data of the input cache 306) are stored in the DRAM bank 302 to optimize for model performance, power consumption, memory consumption, and / or computational efficiency. In this example, the weights of the neural network model are stored in the DRAM bank 302 with a data type of FixedINT4 (4 bits).

[0054] A hybrid quantization logic module 308 receives data (e.g., weights of the neural network model) corresponding to the cached data of the input cache 306. The hybrid quantization logic module 308 converts the data of the input cache 306, which is represented by a datatype of FixedINT4, into a datatype of FixedINT8, to accommodate the particular parameters of multiple ALUs of the system. In this case, the hybrid quantization logic module 308 decomposes the data of the input cache 306 that represents 256 bits of data into 64 sections of 4 bit data elements. In other words, the 256 bits of data of the input cache 306 is split into 64 integers associated with the weights of the neural network model, where each of the weights is represented by a 4 bit (e.g., FixedINT4) integer. The hybrid quantization logic module 308 converts each FixedINT4 into FixedINT8.

[0055] In some implementations, the hybrid quantization logic module 308 converts the multiple FixedINT4 weights to respective FixedINT8 weights to accommodate computational requirements of multiple ALUs that require input data with data type of FixedINT8. In some other implementations, the one or more ALUs require input data with data type of FixedINT16, FixedINT4, etc., and the system configures an appropriate hybrid quantization logic module to accommodate the particular design of the one or more ALUs.

[0056] For each of the 64 FixedINT8 weights converted by the hybrid quantization logic module 308, a respective ALU (e.g.. ALU 310) processes the corresponding FixedINT8 weight as a first operand (e.g., the first operand 202 of Fig. 2) along with a FixedINT8 activation (e.g., the second operand 204 of Fig. 2) from an input buffer 312. In some implementations, the input buffer 312 includes output activations from a layer of the neural network model or from the initialization parameters of the neural network model. For example, the input buffer 312 can include activations to be multiplied by the weights corresponding to the weights stored in the DRAM bank 302 for the corresponding layer of the neural network model. Each activation from the input buffer 312 is broadcast across the full set of ALUs (e.g., ALU 310), where each ALU corresponds to one of the 64 outputs of the hybrid quantization logic module 308. Each ALU processes an output from the hybrid quantization logic module 308 as a first operand and an activation from the input buffer 312 as a second operand. In this example, there are 64 ALUs to process the outputs of each of the 64 FixedINT8 outputs of the hybrid quantization logic module 308.

[0057] Each of the 64 ALUs are configured to process FixedINT8 operands. In this example, the activations from the input buffer 312 are represented by a FixedINT8 data type and the outputs of the hybrid quantization logic module 308 are represented with a FixedINT8 data ty pe. The output of each ALU (e.g., ALU 310) is represented by a FixedINT32 data type. A multiplier (e.g., multiplier 210 of Fig. 2) produces an intermediate product (e.g., intermediate product 208 of Fig. 2) of two FixedINT8 numbers with a FixedINT16 data type. An accumulator (e g., accumulator 220 of Fig. 2) adds two FixedINT16 numbers to yield a FixedINT32 output that is stored in a corresponding output register (e.g., output register 314). Each ALU stores the output of a MAC operation in a corresponding output register (e.g., ALU 310 stores a corresponding output in output register 314).

[0058] In a general sense, the hybrid quantization logic module 308 acts as an intermediate data ty pe transformation module between parameters of a neural network model stored in a memory device (e.g., DRAM 302) and a set of multiple ALUs that are configured to process input data with a particular data type.

[0059] Fig. 4 is a block diagram of an example computing system that implements hybrid quantization logic and area-efficient hardware quantization. A hybrid quantization logic module 408 retrieves weights (e.g., model parameters) from an input cache 406 similar to the system described in relation to Fig. 3. A DRAM bank 402 stores weights associated with a neural network model in multiple sections, pages, columns, or rows (e.g., memory section404 that includes 32B of data). In some cases, the DRAM bank 402 is part of a PIM (e.g., the memory device 122) that is performing the MAC operations associated with the hybrid quantization logic and the area-efficient hardware quantization.

[0060] The hybrid quantization logic module 408 includes the functionality described in relation to Fig. 3, in which weights of the neural network model that are stored in the DRAM bank 402 are represented with a FixedINT4 data type. The hybrid quantization logic module 408 converts the input FixedINT4 weights into corresponding FixedINT8 weights.

[0061] In this example, an area-efficient zero-point subtraction module 410 processes the outputs of the hybrid quantization logic module 408. Each FixedINT8 output of the hybrid quantization logic module 408 (e.g., the 64 output integers that are processed by corresponding ALUs in reference to Fig. 3). is transformed by a zero-point subtraction operation. The details of the zero-point subtraction operation are discussed in detail in relation to Fig. 6. In summary, the area-efficient zero-point subtraction module 410 processes each FixedNT8 operand to generate a 9-bit signed integer (e.g., SINT9).

[0062] The area-efficient zero-point subtraction module 410 generates a corresponding SINT9 operand for each operand it receives from the hybrid quantization logic module 408 and an input buffer (e.g., the input buffer 312) that stores activations (e.g., activation 416). An area-efficient hardware quantization module 412 processes each SINT9 operand from the area-efficient zero-point subtraction module 410. In contrast to the hybrid quantization logic module 408, the area-efficient hardware quantization module 412 provides support for multiple input data types, e g., parameters stored in the DRAM bank 402 as FixedINT2, FixedINTI, etc. Further details describing the specific components of the area-efficient hardware quantization module 412 are described in relation to FIG. 5.

[0063] In summary’, the outputs of the area-efficient hardware quantization module 412 are stored in corresponding output registers (e.g., output register 414). For example, a particular weight stored in the DRAM bank 402 (e.g., in the memory section 404) as a FixedINT4 integer is first received from the input cache 406 by the hybrid quantization logic module 408. The hybrid quantization logic module 408 converts the input FixedINT4 weight to a FixedINT8 weight. The area-efficient zero-point subtraction module 410 converts the FixedINT8 weight to a SINT9 weight. The area-efficient hardware quantization module 412 processes the SINT9 weight and the SINT9 activation 416 to generate an output of a MAC operation performed by one or more ALUs and stores the result of the MAC operation in the associated output register 414. A corresponding process is executed for each weight stored in the DRAM bank 402.

[0064] In some implementations, the logic executed in relation to the hybrid quantization logic module 412 can occur after the logic executed in relation to the area-efficient hardware quantization module 412.

[0065] Fig. 5 is a block diagram of an example computing system that implements areaefficient hardware quantization. An area-efficient hardware quantization module 504 processes output signed integers from an area-efficient zero-point subtraction module 502. In some implementations, the area-efficient zero-point subtraction module 502 processes weights from a hybrid quantization logic module (e.g., the hybrid quantization logic module 408) and activations from an input buffer (e.g., the input buffer 312).

[0066] The system illustrated in Fig. 5 represents four scenarios that are supported by the area-efficient hardware quantization module 504. A first scenario 550 represents a MAC operation that includes a first operand 510 represented by a FixedINT8 number (e.g., an activation) and a second operand 512 represented by a FixedINT8 parameter (e.g., a model weight stored in memory). A second scenario 560 represents MAC operations that includes two pairs of operands, each pair including an operand represented by a FixedINT4 activation and a FixedINT4 parameter. The first pair includes a first operand 510 and a second operand 512, and the second pair includes a first operand 514 and a second operand 516. In contrast to the scenario 550, the first operands 510-512 are represented by FixedINT4 numbers.

[0067] Two additional scenarios are illustrated that represent MAC operations between FixedINT2 numbers and FixedINTI numbers respectively. A third scenario 570 represents MAC operations that includes four pairs of operands, each pair including an operand represented by a FixedINT2 activation and a FixedINT2 parameter. A fourth scenario 580 represents MAC operations that includes eight pairs of operands, each pair including an operand represented by a FixedINTI activation and a FixedINTI parameter. Each scenario provides support for MAC operations between operands of a particular data type.

[0068] In relation to the first scenario 550, the zero-point subtraction module 502 converts each operand (e.g., the first operand 510 and the second operand 512) into respective SINT9 integers. A multiplier 520 multiplies the first operand 510 (SINT9) with the second operand 512 (SINT9) to generate an intermediate product with a data type of SINT17. A multiplexer 522 receives the intermediate product from the multiplier 520 and outputs a result from one of multiple scenarios (e.g., a second scenario 560), which will be described in detail below. An accumulator 524 receives the output from the MUX 522 and updates a running sum according to a sequence of particular MAC operations. The output of theaccumulator has a data type of SINT32. The accumulator 524 may be referred to as an accumulating addition unit.

[0069] The second scenario 560 represents a MAC operation that supports operand data types of FixedINT4. Two pairs of operands are included in the MAC operation. A first pair of operands includes a first operand 510 represented by a FixedINT4 number and a second operand 512 represented by a FixedINT4 number. A second pair of operands includes a first operand 514 represented by a FixedINT4 number and a second operand 516 represented by a FixedINT4 number. In some implementations, an initial FixedINT8 number that includes two stacked FixedINT4 parameters is split into the two corresponding FixedINT4 parameters (e.g., the first parameter 510 and the first parameter 514). Similarly, an initial FixedINT8 number that includes two stacked FixedINT4 activations is split into the two corresponding FixedINT4 activations (e.g., the second operand 512 and the second operand 516).

[0070] The zero-point subtraction module 502 converts each FixedINT4 operand (operands 510-516 associated with the second scenario 560) to a respective signed integer SINT5. In this scenario, the zero-point subtraction module 502 converts each parameter of data type FixedINT4 to SINT5. In some implementations, a zero-point value is subtracted from each FixedINT4 operand to shift the zero-point such that a positive integer value is mapped onto a range of values w ith both positive and negative integers.

[0071] Two multipliers 520 and 532 perform respective multiplication operations between the first pair of operands 510-512 and the second pair of operands 514-516. The multiplier 520 is the same multiplier described in relation to the first scenario 550, in which the multiplier 520 can support multiplication of tw o SINT9 numbers. How ever, in the second scenario 560. the first pair of operands are of FixedINT4. In this case, the multiplier 520 can accommodate SINT5 operands as well because the SINT5 data type is less precise than the SINT9 data type. In other words, the multiplier 520 is used for operations in relation to both the first scenario 550 and the second scenario 560.

[0072] An accumulator 536 adds the intermediate sums from the multiplier 520 and the multiplier 532. The multiplexer 522 receives the sum of the intermediate products generated by the multiplier 520 and the multiplier 532 and outputs the sum to the accumulator 524.

[0073] As an illustrative example, consider an example use case of the second scenario 560. A first FixedINT8 number 10110110 represents two stacked FixedINT4 parameters of a neural netw ork model. The two stacked FixedINT4 parameters represent a decimal value of 11 (e.g., a first operand 510) and a decimal value of 6 (e.g., a first operand 514). To further the example, consider a second FixedINT8 number 11100111, which represents tw o stackedFixedINT4 activations. The two stacked FixedINT4 activations represent a decimal value of 14 (e.g., a second operand 512) and a decimal value of 7 (e.g., a second operand 516). The zero-point subtraction module 502 converts each operand to a SINT5 data type. The respective signed integers are 01011, 00110, and 01110, and 00111 where a leading 0 indicates a positive integer, in which the zero-point value is assumed to be 0. In some implementations, the zero-point value can be defined as a value other than 0.

[0074] The multiplier 520. which is configured to support a multiplication of integers up to SINT9, multiples the SINT5 first operand 510 (01011 = +11) with the SINT5 second operand 512 (01110 = +14) to yield a first intermediate product of 010011010, which represents a decimal value of +154. The multiplier 532, which is an additional hardware component included to support the FixedINT4 x FixedINT4 case (the second scenario 560), is configured to support a multiplication up to SINT5. The multiplier 532 multiplies the SINT5 first operand 514 (00110 = +6) with the SINT5 second operand 516 (00111 = +7) to yield a second intermediate product of 000101010, which represents a decimal value of +42.

[0075] An accumulator 536 is configured to add values up to a bit width of SINT9. The accumulator 536 processes the intermediate product generated by the multiplier 520, which is represented by a SINT9 value of 010011010, and the intermediate product generated by the multiplier 532, which is represented by a SINT9 value of 000101010. The sum of the two SINT9 intermediate products results in a SINT10 value of 010011010 + 000101010 = 011000100, which represents a decimal value of 196.

[0076] The output of the accumulator 536 is processed by the MUX 522, which is shared with the first scenario 550, and all additional scenarios that process additional combination of data types between a first input corresponding to a parameter of a machine learning model stored in a memory device and a second input corresponding to an activation input (e.g., the scenario 570 representing the FixedINT2 x FixedINT2 case and the scenario 580 representing the FixedINTI x FixedINTI case).

[0077] The two scenarios 550 and 560 represent support for two datatype combinations. The first scenario 550 supports a first operand of FixedINT8 and a second operand of FixedINT8. The second scenario 560 supports a first operand of FixedINT4 and a second operand of FixedINT4. Additional scenarios 570 and 580 support FixedINT2 x FixedINT2 and FixedINTI x FixedINTI respectively and can be designed using an analogous design, reusing appropriate multipliers and accumulators included to support the higher precision data types.

[0078] The third scenario 570 includes four pairs of operands. The first pair of operands includes a first operand 510 and a second operand 512, in which both operands 510 and 512 are represented by FixedINT2 numbers. Similarly, the second pair of operands includes a first operand 514 and a second operand 516, in which both operands 514 and 516 are represented by FixedINT2 numbers. Similarly, two additional pairs of FixedINT2 operands 572 are included in the third scenario 570. To support the FixedINT2 data type, additional ALUs are included, and existing ALUs that support higher data precision are reused. For example, two additional SINT3 multipliers (e.g., a multiplier 574) and an additional SINT5 accumulator 576 and a SINT6 accumulator 578, which receives an input from the accumulator 536, are included to support the FixedINT2 operations.

[0079] The fourth scenario 580 includes eight pairs of operands, each operand represented by a FixedINTI number. The first pair of operands includes a first operand 510 and a second operand 512, in which both operands 510 and 512 are represented by FixedINTI numbers. Similarly, the second pair of operands includes a first operand 514 and a second operand 516, in which both operands 514 and 516 are represented by FixedINTI numbers. Similarly, two pairs of FixedINTI operands 572 are included in the fourth scenario 580 (the pairs of operands 572 were included to support the FixedINT2 x FixedINT2 operation in relation to the third scenario 570). Four additional pairs of FixedINTI operands 582 are included in the fourth scenario 580. To support the FixedINTI data type, additional ALUs are included, and existing ALUs that support higher data precision are reused. For example, simple logic units (e.g., logic unit 584) are included that perform operations between 1 -bit integers (e.g., 1, 0, and -1) and additional accumulators (e.g., accumulators 586, 588, and 589) are included to combine the outputs from the operations that support higher precision data types.

[0080] An advantage of the system described in relation to Fig. 5 is the sharing of multiple resources between scenarios that support MAC operations between operands with different data types. For example, resources contained within boundary 534, e.g., the multiplier 532 and the accumulator 536, are included to support the FixedINT4 x FixedINT4 case of the second scenario 560. However, the multiplier 520 is used to perform the operations in relation to the first scenario 550 and the second scenario 560. In addition to sharing MAC operation resources, the system benefits from shared resources within the zeropoint subtraction module 502 which is discussed in detail in relation to Fig. 6.

[0081] As support is added for lower precision data types, additional resources (e.g.. multipliers and accumulators) are required, but the multipliers and accumulators from thehigher precision data ty pe support scenarios can be shared with the new lower precision data type scenarios. For example, a scenario that supports FixedINT2 x FixedINT2 will share the multiplier 520, multiplier 532, and the accumulator 536, while it introduces two new multipliers and two new accumulators. However, the new multipliers and accumulators will support the lower precision data type, which are more energy and area efficient.

[0082] In some cases, in which area-efficiency is not the driving optimization factor to designing a multi-data type MAC operation system, a dedicated set of multipliers and accumulators may be appropriate. For example, for a system that is optimized for energy efficiency, it is inefficient to re-use high precision data type multipliers (e.g., multiplier 520) for a low precision data type MAC operation (e.g., FixedINT2 x FixedINT2). However, for applications in which area-efficiency is the driving optimization factor (e.g., in some cases, PIM devices), the ability to share resources across data type MAC operations can be appropriate.

[0083] In some implementations, a register of a system that implements the area-efficient hardware quantization indicates a bit width (e.g., data type) of the input operands (e.g.. operand 510 and operand 512). The register may be referred to as a configuration register. The appropriate scenario (e.g., 550 or 560) is selected according to the configured register value. In some implementations, the multiplexer 522 receives a control value from the register. That is, the configuration register may be configured to indicate an input multiplexer control value. The control value is indicative of the data types of the input operands and determines the mode of operation of the multiplexer 522. For example, the control value can configure the multiplexer 522 to output a data value received from the multiplier 520 if the system is configured to operate on FixedINT8 operands (e.g., FixedINT8 operand 510 and FixedINT8 operand 512). As another example, the control value can configure the multiplexer 522 to output a data value received from the adder 536 if the system is configured to operate on FixedINT4 operands (e.g., FixedINT4 operand 510, FixedINT4 operand 512, FixedINT4 operand 514, and FixedINT4 operand 516). In some implementations, a user (e.g., a software engineer) can set the values of the registers of the system to determine which data types represent the activation and model parameters.

[0084] Fig. 6 illustrates zero-point subtraction logic. A zero-point subtraction module (e.g., the zero-point subtraction module 502) implements zero-point subtraction logic on input operands that have a particular data type (e.g., FixedINT8, FixedINT4, etc.). An example input to the zero-point subtraction module is a FixedINT8 data type operand (e.g., 10100010). An example output of the zero-point subtraction module is a SINT9 data typenumber (e.g.. 010100010), where a leading 0 is added to the FixedINT8 to indicate the positive sign. In some implementations, the zero-point subtraction logic subtracts a FixedINT8 number from the input operand to shift the zero point such that the output is mapped to a range of positive and negative integers.

[0085] To support multiple input data types, the zero-point subtraction logic includes multiple pairs of addition modules (e.g., pairs of addition modules 640-646), in which each addition module of a pair of addition modules supports a maximum bit-width data type. For example, the pair of addition modules 640 support addition of numbers up to FixedINT8, the pair of addition modules 642 support addition of numbers up to FixedINT4, and addition modules 644-646 support addition of numbers up to FixedINT2. In some cases, addition modules that support larger bit widths (e.g., the pair of addition modules 640) consume more power and are potentially larger than the smaller bit width (e.g., pair of addition modules 646) counterparts.

[0086] The zero-point subtraction logic illustrated in Fig. 6 includes three examples 602- 606 of a first input data type (e.g., first input 610) and a second input data type (e.g., second input 612) that are transformed by a zero-point subtraction process by one or more addition modules (e.g., the pairs of addition modules 640-646). The first example 602 illustrates zeropoint subtraction logic executed on a first input of FixedINT8 610 (activation) and a second input of FixedINT8 612 (neural network w eight). A pair of first addition modules 640 that support addition of numbers up to FixedINT8 perform the zero-point subtraction for the first input 610 and the second input 612.

[0087] The second example 604 illustrates zero-point subtraction logic executed on two sets of numbers of FixedINT4. This example corresponds to scenario 560 as described in relation to FIG. 5. In this case, a FixedINT8 binary representation is split into a FixedINT4 first operand and a FixedINT4 second operand, where each operand corresponds to a particular FixedINT4 activation. The model weights stored in memory are represented as FixedINT4 as well. The zero-point subtraction logic illustrated in this example performs two addition operations. The first pair of addition modules 640 perform the zero-point subtraction for a first input of FixedINT4 614 and a second input of FixedINT4 616. A second pair of addition modules 642 perform a zero-point subtraction for a third input of FixedINT4 618 and a fourth input of FixedINT4 640. In this example, the first input of FixedINT4 614 and the third input of FixedINT4 618 represent the first operand and the second operand of the stacked activation input. The second input of FixedINT4 616 and the fourth input ofFixeINT4 620 represent the weight values that correspond to weights of a particular neural network model.

[0088] The third example 606 illustrates zero-point subtraction logic executed on four sets of numbers of FixedINT2. This example is a continuation of the same logic as described in relation to the second example 604. The extension of the logic to support zero-point subtraction for FixedINT2 includes two additional pairs of addition modules 644-646. The addition modules 640-642 described in relation to the first example 602 and the second example 604 are reused to support of the zero-point subtraction logic for FixedINT2 numbers. As a further extrapolation, the logic can be extended to support FixedINTI by including four additional addition modules to support the conversion of additional pairs of numbers.

[0089] In some implementations, the zero-point subtraction logic described in relation to Fig. 6 is an area-efficient zero-point subtraction logic because the support for multiple data types includes hardware resource sharing. For example, the pair of accumulators 640 are used to support the examples 602-606, in which each example describes a zero-point subtraction logic for a different data type. Similar to the example described in relation to Fig. 5, as support for lower precision data ty pes is included, additional hardware components are required (e.g., more accumulators), but the new hardware components need not require high precision data types.|0090| Fig. 7 illustrates an example process 700 that implements area-efficient hardware quantization. For convenience, the process will be described as being performed by a system configured to operate in accordance with this specification.

[0091] The system performs (702) a first multiplication between input operands of a first input data using a first multiply unit. The input operands of the first input data have a first bit width. The first multiply unit is operable to multiply input operands with a bit width greater than the first bit width. For example, the first input data can include a first operand associated with an activation value and a second operand associated with a neural network model weight. In this example, each operand can be described by a bit width of FixedINT4 and the first multiply unit can accommodate a multiplication of operands with data values up to FixedINT8. This example corresponds to the first scenario 550 as described in relation to Fig. 5, in which the input operands are represented as FixedINT4, and the first multiply unit can accommodate input operands up to FixedINT8 (greater than FixedINT4).

[0092] The system performs (704) a second multiplication between input operands of a second input data using a second multiply unit. The input operands of the second input datahave the first bit width, and the second multiply unit is operable to multiply input operands with a bit width less than or equal to the first bit width. For example, the second input data can include a first operand associated with an activation value and a second operand associated with a neural network model w eight, similar to the example described above in relation to (702). In this example, each operand can be described by a bit width of FixedINT4 (the same bit width as the input operands of the first input data). However, in this example, the second multiply unit is operable to multiply input operands with a bit width up to and including the first bit width, rather than bit widths greater than the first bit width, as is the case for the first multiply unit. This example corresponds to the first scenario 550 as described in relation to Fig. 5, in which the input operands are represented as FixedINT4, and the second multiply unit can accommodate input operands up to FixedINT4 (but not greater).

[0093] The system adds (706) a result of the first multiply unit and the second multiply unit with a first addition unit operable to add input operands with a bit width greater than the first bit width. For example, the result of the first multiply unit can be associated with an output of a multiply unit that multiples input operands of FixedINT4 bit width but can accommodate input operands up to FixedINT8 (e.g., multiplier 520). The result of the second multiply unit can be associated with an output of a multiply unit that multiplies input operands of FixedINT4 bit width and can only accommodate input operands up to FixedINT4 (e.g., multiplier 532). The first addition unit (e.g., accumulator 536) adds the result of the first multiply unit and the result of the second multiply unit and can accommodate bit widths greater than the first bit width (e.g., up to bit width of FixedINT8) because it accepts output from the multiply unit that can accommodate input operands up to FixedINT8.

[0094] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g.. code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0095] A computer program (w hich may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0096] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0097] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry7, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0098] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random-access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0099] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0100] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0101] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. These embodiments can include processes and methods of making or manufacturing a semiconductor device, a wafer-level package, packaging features, or package structures described in this specification.

[0102] Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0103] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0104] In addition to the embodiments described above, the following embodiments are also innovative:

[0105] Embodiment 1 is a multiply accumulate device comprising:

[0106] a first multiply unit having a first bit width; and

[0107] a second multiply unit having a second bit width that is smaller than the first bit width.

[0108] w herein the device is reconfigurable to perform:

[0109] a first multiplication operation using the first multiply unit when input data comprises input operands having the first bit width, and

[0110] a plurality of second multiplication operations using the first multiply unit and the second multiply unit when input data comprises input operands having the second bit width.

[0111] Embodiment 2 is the device of embodiment 1, further comprising:

[0112] an addition unit configured to:

[0113] accumulate a result of the first multiplication operation when the input data comprises input operands having the first bit width, or

[0114] accumulate a sum of results of the plurality of second multiplication operations when the input data comprises input operands having the second bit width.

[0115] Embodiment 3 is the device of embodiment 2, further comprising a second addition unit configured to generate the sum of results of the plurality of second multiplication operations by adding the result of the first multiply unit and the second multiply unit.

[0116] Embodiment 4 is the device of any of embodiments 1-3. further comprising:|00117| a third multiply unit having a third bit width that is smaller than the second bit width,

[0118] wherein the device is reconfigurable to perform:

[0119] a plurality of third multiplication operations using the first multiply unit, the second multiply unit, and the third multiply unit when input data comprises input operands having the third bit width.

[0120] Embodiment 5 is the device of embodiment 4, further comprising:

[0121] an additional third multiply unit having the third bit width that is smaller than the second bit width.

[0122] wherein the device is reconfigurable to perform:

[0123] the plurality of third multiplication operations using the first multiply unit, the second multiply unit, the third multiply unit and the additional third multiply unit w hen input data comprises input operands having the third bit w idth.

[0124] Embodiment 6 is the device of embodiment 5. further comprising a third addition unit configured to generate a sum of results of the third multiply unit and the additional thirdmultiply unit, and a fourth addition unit configured to generate a sum of results of the second addition unit and the third addition unit.

[0125] Embodiment 7 is the device of any of embodiments 1-6, further comprising:

[0126] a plurality of fourth multiply units that are configured to operate on 1 -bit input operands; and

[0127] a plurality of addition units configured to collectively sum a result of the plurality of fourth multiply units.

[0128] Embodiment 8 is the device of embodiment 7 and embodiment 6, further comprising a fifth addition unit configured to generate a sum of results of the fourth addition unit and the plurality of addition units.

[0129] Embodiment 9 is the device of any of embodiments 1-8. further comprising a configuration register configured to indicate a bit width of the input operands.

[0130] Embodiment 10 is the device of any of embodiments 1-9, wherein the input operands comprise pairs of activation values and parameter values of a machine learning model.

[0131] Embodiment 11 is the device of any of embodiments 1-10, wherein the device is a processing element of a processing-in-memory architecture.

[0132] Embodiment 12 is the device of any of embodiments 1-11, further comprising a zero-point subtraction unit configured to convert each operand of the input data from a fixed integer data value to a signed integer data value.

[0133] Embodiment 13 is the device of any of embodiments 1-12, further comprising a multiplexer operable to receive an output from the first multiply unit and an output from the sum of results of the plurality of second multiplication operations, wherein the multiplexer is configurable to provide an output to the addition unit based on an input multiplexer control value.

[0134] Embodiment 14 is the device of embodiment 13 and embodiment 3, wherein the multiplexer is operable to receive an output from the second addition unit.

[0135] Embodiment 15 is the device of embodiment 14 and embodiment 6, wherein the multiplexer is operable to receive an output from the fourth addition unit.

[0136] Embodiment 16 is the device of embodiment 15 and embodiment 7, wherein the multiplexer is operable to receive an output from the plurality of addition units.

[0137] Embodiment 17 is the device of embodiment 15 and embodiment 8, wherein the multiplexer is operable to receive an output from the fifth addition unit.

[0138] Embodiment 18 is the device of any of embodiments 13-17, further comprising a configuration register configured to indicate the input multiplexer control value.

[0139] Embodiment 19 is a method performed by a multiply accumulate device, the method comprising:

[0140] performing a first multiplication between input operands of a first input data using a first multiply unit, the input operands of the first input data having a first bit width, the first multiply unit operable to multiply input operands with a bit width greater than the first bit width:

[0141] performing a second multiplication between input operands of a second input data using a second multiply unit, the input operands of the second input data having the first bit width, the second multiply unit operable to multiply input operands with a bit width less than or equal to the first bit width; and

[0142] adding a result of the first multiply unit and the second multiply unit with a first addition unit operable to add input operands with a bit width greater than the first bit width.

[0143] Embodiment 20 is the method of embodiment 19, further comprising:

[0144] accumulating a result of the first addition unit with an accumulating addition unit.

[0145] Embodiment 21 is the method of any of embodiments 19-20, wherein a value stored in a configuration register is indicative of the first bit width of the input operands of the first input data.|00146| Embodiment 22 is the method of any of embodiments 19-21, wherein the input operands comprise pairs of activation values and parameter values of a machine learning model.

[0147] Embodiment 23 is the method of any of embodiments 19-22, wherein the device is a processing element of a processing-in-memory architecture.

[0148] Embodiment 24 is the method of any of embodiments 19-23, further comprising performing a zero-point subtraction that converts each operand of each input data from a fixed integer data value to a signed integer data value.

[0149] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0150] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0151] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:

1. A multiply accumulate device comprising: a first multiply unit having a first bit width; and a second multiply unit having a second bit width that is smaller than the first bit width, wherein the device is reconfigurable to perform: a first multiplication operation using the first multiply unit when input data comprises input operands having the first bit width, and a plurality of second multiplication operations using the first multiply unit and the second multiply unit when input data comprises input operands having the second bit width.

2. The device of claim 1, further comprising: an addition unit configured to: accumulate a result of the first multiplication operation when the input data comprises input operands having the first bit width, or accumulate a sum of results of the plurality of second multiplication operations when the input data comprises input operands having the second bit width.

3. The device of any of claims 1-2, further comprising: a third multiply unit having a third bit width that is smaller than the second bit width, wherein the device is reconfigurable to perform: a plurality of third multiplication operations using the first multiply unit, the second multiply unit, and the third multiply unit when input data comprises input operands having the third bit width.

4. The device of any of claims 1-3, further comprising: a plurality of fourth multiply units that are configured to operate on 1 -bit input operands; and a plurality of addition units configured to collectively sum a result of the plurality of fourth multiply units.

5. The device of any of claims 1-4, further comprising a configuration register configured to indicate the bit width of the input operands.

6. The device of any of claims 1-5, wherein the input operands comprise pairs of activation values and parameter values of a machine learning model.

7. The device of any of claims 1-6. wherein the device is a processing element of a processing-in-memory architecture.

8. The device of any of claims 1-7, further comprising a zero-point subtraction unit configured to convert each operand of the input data from a fixed integer data value to a signed integer data value.

9. The device of any of claim 2-8, or any preceding claim dependent thereon, further comprising a multiplexer operable to receive an output from the first multiply unit and an output from the sum of results of the plurality of second multiplication operations, wherein the multiplexer is configurable to provide an output to the addition unit based on an input multiplexer control value.

10. The device of any of claims 1-9. further comprising a configuration register configured to indicate the input multiplexer control value.

11. A method performed by a multiply accumulate device, the method comprising: performing a first multiplication between input operands of a first input data using a first multiply unit, the input operands of the first input data having a first bit width, the first multiply unit operable to multiply input operands with a bit width greater than the first bit width; performing a second multiplication between input operands of a second input data using a second multiply unit, the input operands of the second input data having the first bit width, the second multiply unit operable to multiply input operands with a bit width less than or equal to the first bit width; and adding a result of the first multiply unit and the second multiply unit with a first addition unit operable to add input operands with a bit width greater than the first bit width.

12. The method of claim 11, further comprising: accumulating a result of the first addition unit with an accumulating addition unit.

13. The method of any of claims 11-12, wherein a value stored in a configuration register is indicative of the first bit width of the input operands of the first input data.

14. The method of any of claims 11-13, wherein the input operands comprise pairs of activation values and parameter values of a machine learning model.

15. The method of any of claims 11-14, wherein the device is a processing element of a processing-in-memory architecture.

16. The method of any of claims 11-15, further comprising performing a zero-point subtraction that converts each operand of each input data from a fixed integer data value to a signed integer data value.

Images