Exploiting input and output sparsity for in-memory machine-learning computations

Abstract

Methods and systems, including computational instructions / programs encoded on computer-readable media, are described for exploiting input and output sparsity for in-memory machine-learning computations. A Processing-in-Memory ("PiM") architecture for an integrated memory device includes compute units, a shared queue coupled to each compute unit, reduction circuitry, and a shared datapath. The queue stores (i) inputs that are routed along the shared datapath for processing by the compute units and (ii) outputs that are generated after the compute units process the inputs. Each compute unit includes arithmetic circuitry for computing accumulated values from the inputs. The reduction circuitry generates the outputs based on reduction operations performed on accumulated values received from the compute units. The outputs are routed to the queue for storage using a portion of the shared datapath that couples the reduction circuitry to the shared queue.

Description

Attorney Docket No. 56113-0798WO1EXPLOITING INPUT AND OUTPUT SPARSITY FOR IN-MEMORY MACHINE-LEARNING COMPUTATIONSCLAIM OF PRIORITY

[0001] This application claims priority under 35 USC §119(e) to U.S. Patent Application Serial No. 63 / 680.043, filed on August 6. 2024, the entire contents of which are hereby incorporated by reference.BACKGROUND

[0002] This specification relates to performing machine-learning computations using a special-purpose hardware computing unit.

[0003] Modem computing systems often incorporate a wide variety of computer processing units that each offer different computing capabilities and trade-offs. Efficient execution of a given compute job often involves parsing computations into meaningful subtasks or workloads that are mapped to available processor cores of a computing system. The computations may be parsed and mapped based on suitability criteria, such as processor capability, performance, and power. Generally, this overall process of allocating portions of a computation to appropriate processor resources is referred to as heterogeneous computing.

[0004] At least one processor core of a System-on-Chip (“SoC”) can be an Intellectual Property block (“IP block’7) that executes a respective portion of a computational operation for different multimedia workloads. Example use cases can involve processing image or speech data captured respectively by a camera or microphone on the mobile device as well as performing computations for generative artificial intelligence (“GenAI"’) applications. The system-on-chip can use a heterogeneous compute operation to process input samples derived from image data, speech data, a text corpus, or a combination of these. An example step in the heterogeneous compute operation can include processing data associated with the input samples using a memory device that provides in-memory processing or computing capabilities.SUMMARY

[0005] Example machine-learning computations, such as computing convolutions, often require numerous matrix multiplications in a large dimensional space, which can be performed in hardware using a computational unit of a special-purpose hardware integrated circuit, such as a neural network processor, tensor processor, or hardware accelerator. A processor or hardware controller of the integrated circuit can pass control signals to theAttorney Docket No. 56113-0798WO1 computational unit to execute compute-intensive matrix multiplications that involve repeatedly calculating products and partial sums for the convolutions. The degree to which the processor parallelizes calculations is limited due to its architecture.

[0006] This specification describes hardware and software architecture(s) and associated techniques that enable exploitation of input and output sparsity within a Processing-in- Memory ("PiM") architecture of an integrated memory device, such as a dynamic randomaccess memory device (“DRAM”). The architecture includes shared buffers and at least one data path that is configured for both input broadcasting and output reduction within the PiM architecture. The disclosed techniques, and corresponding PiM architecture, can also include an input bus / path that couples an input queue of the PiM architecture to arithmetic circuitry of a computational unit in the PiM architecture. The input bus / path is configured to be shared for both broadcasting of inputs and unicasting of inputs (e.g., machine-learning inputs) to multiply-accumulate cells (MACs) that represent arithmetic circuitry of the computational unit in the PIM architecture.

[0007] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0008] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages. Techniques are described for exploiting input and output sparsity in data processed by executing machine-learning computations using resources of a PiM architecture in an integrated circuit of a memory device. The disclosed PiM architecture moves machine learning computations directly into memory, thereby overcoming limitations of traditional processor architectures. For example, the disclosed system exploits input and output sparsity’ through a unified, shared data path for both input broadcasting and output reduction, along with adaptive input handling via hardware multiplexers. Dedicated reduction circuitry (e g., adder trees) within the memory’ device, enables more efficient processing of large matrix multiplications inherent in machine learning, reducing data movement bottlenecks, and enhancing overall performance beyond what is achievable with general-purpose processors.Attorney Docket No. 56113-0798WO1

[0009] Operations of the described system results in reduced data transmission between a processor and external memory, as computations are performed in-memory. Furthermore, the described system allows for enhanced parallelization for compute-intensive matrix multiplications, as operations can be implemented where the data is stored, rather than a shared processor. Furthermore, the sharing of the input data path and the output data path provides a reduction in hardware area overhead.

[0010] In an aspect, a PiM architecture for a memory device includes a shared queue configured to store a set of inputs and one or more outputs, multiple compute units, each of the multiple compute units including arithmetic circuitry for computing accumulated values from one or more inputs in the set of inputs, reduction circuitry configured to generate the one or more outputs based on reduction operations performed on the accumulated values, and a shared datapath configured to (i) provide the one or more inputs to the multiple compute units, (ii) provide the accumulated values to the reduction circuitry, and(iii) provide the one or more outputs to the shared queue.

[0011] In some implementations, the shared datapath couples the shared queue to each of the multiple compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture.

[0012] In some implementations, the shared datapath is configured as a unified datapath that is shared between at least two distinct modes of the PiM architecture.

[0013] In some implementations, the at least two distinct modes of the PiM architecture include (i) an input broadcast mode where one or more inputs are broadcast to each of the multiple compute units and (ii) a reduction mode where one or more outputs are generated based on the reduction operations. In some implementations, the at least two distinct modes of the PiM architecture includes an input unicast mode where one or more inputs are routed to a particular compute unit of the multiple compute units.

[0014] In some implementations, each of the multiple compute units includes a register file configured to buffer input data for a given compute operation and selection logic including a hardware multiplexer configured to (i) route data provided along the shared datapath from the shared queue to the arithmetic circuitry of the compute unit and (ii) route data from the register file to the arithmetic circuitry of the compute unit.

[0015] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to distribute the one or more inputs to the compute unit during each of an input broadcast mode of the PiM architecture and an input unicast mode of the PiM architecture.Attorney Docket No. 56113-0798WO1

[0016] In some implementations, each hardware multiplexer in each respective selection logic is configured to enable adaptive input handling at the PiM architecture based on a sparsity-exploitation mode of the PiM architecture.

[0017] In some implementations, the adaptive input handling includes (i) broadcasting inputs routed from the shared queue and (ii) unicasting input data from a data buffer of the register file of the corresponding compute unit.

[0018] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to (i) receive a select bit for its hardware multiplexer, (ii) select, by its hardware multiplexer, a particular input data source based on the select bit, and (iii) route input data to the arithmetic circuitry of the compute unit based on the particular input data source that is selected based on select bit.

[0019] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to (i) select input data buffered in its register file as the particular input data source based on a first value of the select bit, and (ii) route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.

[0020] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to (i) select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit and (ii) route, from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

[0021] In some implementations, the reduction circuitry includes a first circuit that is a hardware reduction block comprising an adder tree configured to perform accumulation operations on partial sums generated by one or more of the multiple compute units.

[0022] In some implementations, the reduction circuitry includes a second circuit that includes a respective hardware adder tree for each set of compute units that are disposed adjacent, and coupled to, a corresponding bank group of the memory device and each respective hardware adder tree is configured to perform accumulation operations on partial sums generated by the set of compute units to which the respective hardware adder is coupled.

[0023] In some implementations, the PiM architecture includes a bank group controller configured to generate control signaling used to route one or more inputs to particular compute units of the multiple compute units.

[0024] In an aspect, a method performed using a PiM architecture of an integrated memory device includes storing a set of inputs in a shared queue of the PiM architecture.Attorney Docket No. 56113-0798WO1 providing, by way of a shared datapath of the PiM architecture, one or more inputs in the set of inputs from the shared queue to multiple compute units in the PiM architecture, computing, by the multiple compute units, accumulated values from the one or more inputs, providing, via the shared datapath, the accumulated values to reduction circuitry in the PiM architecture configured to perform reduction operations on the accumulated values, generating a machine-learning output from the accumulated values based on the reduction operations, and storing the machine-learning output in the shared queue in response to routing the machine-learning output to the shared queue using the shared datapath.

[0025] In some implementations, the shared datapath couples the shared queue to each of the multiple compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture.

[0026] In some implementations, the shared datapath is configured as a unified datapath that is shared between at least two distinct modes of the PiM architecture.

[0027] In some implementations, the at least two distinct modes of the PiM architecture include (i) an input broadcast mode where one or more inputs are broadcast to each of the multiple compute units and (ii) a reduction mode where one or more outputs are generated based on the reduction operations. In some implementations, the at least two distinct modes of the PiM architecture includes an input unicast mode where one or more inputs are routed to a particular compute unit of the multiple compute units.

[0028] In some implementations, the method includes buffering, by a register file of each compute unit of the multiple compute units, input data for a given compute operation, routing, by a hardware multiplexer of each compute unit configured to perform a respective selection logic, data provided along the shared datapath from the shared queue to the arithmetic circuitry of the compute unit, and routing, by the hardware multiplexer, data from the register file to the arithmetic circuitry of the compute unit.

[0029] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to distribute the one or more inputs to the compute unit during each of an input broadcast mode of the PiM architecture and an input unicast mode of the PiM architecture.

[0030] In some implementations, each hardware multiplexer in each respective selection logic is configured to enable adaptive input handling at the PiM architecture based on a sparsity-exploitation mode of the PiM architecture.Attorney Docket No. 56113-0798WO1

[0031] In some implementations, the adaptive input handling includes (i) broadcasting inputs routed from the shared queue and (ii) unicasting input data from a data buffer of the register file of the corresponding compute unit.

[0032] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to (i) receive a select bit for its hardware multiplexer, (ii) select, by its hardware multiplexer, a particular input data source based on the select bit; and (iii) route input data to the arithmetic circuitry of the compute unit based on the particular input data source that is selected based on select bit.

[0033] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to (i) select input data buffered in its register file as the particular input data source based on a first value of the select bit and (ii) route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.

[0034] In some implementations, each respective selection logic in a compute unit of the multiple compute units is configured to (i) select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit and (ii) route, from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

[0035] In some implementations, the reduction circuitry includes a first circuit that is a hardware reduction block comprising an adder tree configured to perform accumulation operations on partial sums generated by one or more of the plurality of compute units.

[0036] In some implementations, the reduction circuitry includes a second circuit that includes a respective hardware adder tree for each set of compute units that are disposed adjacent, and coupled to. a corresponding bank group of the memory device and each respective hardware adder tree is configured to perform accumulation operations on partial sums generated by the set of compute units to w hich the respective hardware adder is coupled.

[0037] In some implementations, the method includes generating, by a bank controller, control signaling used to route one or more inputs to particular compute units of the multiple compute units.

[0038] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below; Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.Attorney Docket No. 56113-0798WO1BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Fig. 1 is a block diagram of an example computing system with at least one SoC and integrated memory device.

[0040] Figs. 2A & 2B illustrate examples of PiM architectures for an integrated memory device.

[0041] Fig. 3 shows an example of a parameter tensor with K in N sparsity.

[0042] Fig. 4 illustrates example configuration options of computing resources for routing input data to and obtaining reduced outputs from a reduction circuit in a PiM architecture.

[0043] Fig. 5 illustrates an example configuration that enables data routing and computing resources for exploiting input and output sparsity in a PiM architecture.

[0044] Fig. 6 illustrates an example PiM architecture that includes a shared datapath and shared queue for exploiting both input and output sparsity.

[0045] Fig. 7 illustrates an example input bus / path that is shared between two modes of operation and example features of a computing unit in a PiM architecture.

[0046] Fig. 8 illustrates an example configuration option of a PiM architecture that leverages the shared datapath of at least Fig. 7 to enable element-wise operations for executing machine-learning computations.

[0047] Fig. 9 illustrates an example input / output (I / O) queue that is shared between two modes of operation in a PiM architecture.

[0048] Fig. 10 is an example process for exploiting input and output sparsity for inmemory machine-learning computations executed using a PiM architecture.

[0049] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0050] Fig. 1 is a block diagram of an example computing system 100 that includes a system-on-chip 102 (“SoC 102”). The SoC 102 includes a central processing unit 104 (“CPU 104”), a memory controller 105, a shared memory 106 (“memory 106”), a resource manager 108, and an IP / circuit block 110. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at system 100.

[0051] The CPU 104 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a functionAtorney Docket No. 56113-0798WO1 call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0052] The memory 106 is a system memory, shared memory, or both. In the example of Fig. 1, memory7106 is depicted external to circuit block 110. However, memory7106 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory of the SoC 102. such as static random-access memory (SRAM), dynamic random-access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0053] In some implementations, aspects of memory7106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block 110. The memory 106 can also include various other types of memory, such as high bandwidth memory (HBM), narrow memory7(e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0054] The resource manager 108 is implemented in hardware and software. Aspects of the resource manager 108 can be also implemented as firmware of the SoC 102 or firmware of a device of the SoC 102, such as a DRAM memory7device or the CPU 104. The resource manager 108 is a processor-in-memory7(PiM) resource manager (“PiM resource manager 108”) that includes control logic implemented in hardware, software, or both. For example, the PiM resource manager 108 can include resources such as flip-flops, registers, buffers, etc. that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software.

[0055] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include an image signal processor (ISP) 112, a host (or special purpose) processing unit (HPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, HPU 114, DSP 116, and GPU 118 can be a respective proprietary7IP block (or IP device) of a particular entity or device manufacturer.Attorney Docket No. 56113-0798WO1

[0056] One or more aspects of the PiM resource manager 108 can be implemented as a software routine (or module) of the CPU 104, which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 102, such as memory7106. In some implementations, each processor (e.g., ISP 112, DSP 116, HPU 114, GPU 118) of the SoC 102 includes multiple cores and the CPU 104 and / or the PiM resource manager 108 can generate control signaling 124 to manage and distribute memory intensive compute operations to a memory device 122 (e g., DRAM) to minimize the processing load at each core of the processors. The control signaling 124 is routed at system 100 using an example bus 120 of the SoC 102. The control signaling 124 can include commands, requests, data, instructions, or combination of these.

[0057] The PiM resource manager 108 cooperates with the CPU 104, memory controller 105, and / or host processor to dynamically control and manage one or more compute-inmemory7(CiM) operations. In some implementations, the CiM operations are executed at the SoC 102 in support of a heterogeneous compute operation between two or more processing units that are included among the IP block 1 10, the CPU 104. or both. More specifically, the PiM resource manager 108 is configured to generate control signaling 124 and use one or more discrete signal values of the control signaling 124 to initiate and manage PiM operations at the memory device 122.

[0058] The system 100 includes an example memory device 122. The memory device 122 can include multiple memory dies. For example, the memory device 122 can include N memoty die, where N is an integer greater than 1. The memory device 122 can be a dynamic random-access memory7(DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory7device 122 is configured to perform or support various types of PiM operations, CiM operations, and memory-near-computing operations ("MnC operations”). The memory device 122 performs or supports these operations using data processing resources and / or compute elements of its PiM architecture, which are described below with reference to at least Figs. 2A and 2B.

[0059] The SoC 102 cooperates with the memory device 122 to perform computations across one or more bank groups of the memory device 122. The computations can be for operations or workloads that involve one or more of the processors at IP block 110. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In at least one example theAttorney Docket No. 56113-0798WO1 memory' device 122 may be external to the SoC 102, whereas in another example the memory device 122 may be internal to the SoC 102.

[0060] In the example of Fig. 1, system 100 and the SoC 102 is an integrated circuit of an example user / client device 130, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 130a, tablet 130b, laptop 130c, smartwatch or wearable device 130d. The devices 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0061] Fig. 2A and Fig. 2B illustrate examples of PiM architectures for an integrated memory device.

[0062] Fig. 2A shows an example processor-in-memory (PiM) architecture 200 for initiating, controlling, and / or boosting PiM computations and data access performance based on control signals generated using the SoC 102, the memory' device 122, or both. In the example of Fig. 2A, the memory device 122 includes a first memory die-1 with a first bank group that has multiple memory banks, where each memory bank includes one or more memory arrays and a second memory die-2 with a second bank group that has multiple memory' banks, where each memory' bank includes one or more memory' arrays. In some implementations, the PiM architecture 200 includes multiple bank groups, multiple memory die, or both. For example, a single memory die can include multiple bank groups and / or multiple bank groups can be distributed across multiple memory die.

[0063] The PiM architecture 200 includes multiple PiM blocks, where each PiM block includes multiple compute elements. For example, a first PiM block of PiM architecture 200 includes mode register 204-1 and process unit 206-1, whereas a second, different PiM block of PiM architecture 200 includes mode register 204-2 and process unit 206-2. Each process unit 206-1, 206-2 can include a processor, a processor unit, or a processor core, such as a CPU. Each process unit 206-1, 206-2 can also include an example computation unit such as an arithmetic logic unit (ALU) or multiply-accumulate cell (MAC).

[0064] In some implementations, the PiM architecture 200 is included in the memory device 122 as multiple discrete integrated circuits, where each integrated circuit is local to a given memory die (e.g., die-1 and die-2) and interacts or communicates with arrays of memory cells at that memory' die. For example, the PiM architecture 200 can include compute elements that are replicated and distributed across each of the memory die in the memory device 122. In some other implementations, the PiM architecture 200 is included inAttorney Docket No. 56113-0798WO1 the memory' device 122 as a single integrated circuit that interacts or communicates with each memory die of the memory’ device 122, including the arrays of memory cells at each memory die.

[0065] The PiM blocks or process units in the PiM architecture 200 are located within the memory device 122 but outside of a section of the memory device 122 that includes the bank groups. The section may be defined as a discrete memory’ die or defined in some other way (e.g.. a portion of a memory die). Irrespective of the hardware configuration or layout of PiM architecture 200, the PiM blocks are sufficiently external to the bank groups such that the PiM blocks can communicate with the bank groups based on a particular timing constraint that is leveraged to boost PiM data access performance with cross bank group data aggregation. In some implementations, a data channel / interconnection can be established between the individual memory arrays of a memory bank and a corresponding processor unit and / or mode register of a PiM block 202.

[0066] The PiM operations can be managed and executed at the memory device 122 using a processor device that provides functionality similar to a central processor, such as CPU 104. The CiM operations and MnC operations can include standard arithmetic operations, such as computations normally performed by an ALU or MAC. The CiM operations and MnC operations can also include computational functions of a HPU 114, such as multiplication and addition operations for matrix math, vector computations, linear algebra, and dot-product accumulations. In some implementations, each of the PiM operations, CiM operations, and MnC operations are performed in support of machinelearning computations, neural network computations, or both.

[0067] In some implementations, the PiM operations are an extension of the computational functions of the HPU 114. For example, a PiM block can generate accumulated values from sets of weight values / inputs and activation inputs obtained from memory banks of different bank groups in the memory device 122. The accumulated values are generated based on neural network computations performed using a computational array of the PiM block. The computational array can be a matrix multiplication unit with compute cells that are arranged as a systolic array. The accumulated values can be dot products of the sets of weight values and the activation inputs. That is, for a set of weights, the PiM block multiplies each weight with each activation input and sums the products together to form an accumulated value.

[0068] The PiM architecture 200 can include a register or other portion of memory for storing data for a respective memory’ die or group of memory banks. For example, the dataAttorney Docket No. 56113-0798WO1 can be mode / configuration / throtling values. The data can also describe errors that occurred during a compute operation at a corresponding PiM block of the memory device 122, or both. In some implementations, the register or other portion of memory is used to store thermal throttling information, or associated instructions, for configuring aspects of a PiM block, or respective memory die, group of memory7banks, or a combination of these.

[0069] For example, the mode registers 204-1, 204-2 can be used to control or trigger selection of a particular mode in a PiM architecture, such as an error-capture mode, throttle mode, interleave configuration mode, multi-batch processing mode, etc. In some implementations, a particular mode is selected based on bit values of the mode registers 204- 1, 204-2. For example, to trigger or select a thermal throttling mode(s) or multi-batch processing mode(s), a single bit. or a sequence of bits, can be defined for use in the mode register. This is described in more detail below with reference to the example of Fig. 7.

[0070] In some implementations, data access operations can be optimized at the memory device 122 by reading memory cells of bank groups at a frequency that exceeds other read operations that are subject to certain delay constraints for executing successive reads against banks of the memory device 122. For example, an internal controller of the PiM block (1), (2) can execute successive read commands at a frequency that is based on a clock cycle generated by the memory7device 122. The internal controller can operate based on a particular clock frequency, e.g., a 200 or 800 MHz clock or 1000 MHz clock. Other clock frequencies are also within the scope of this disclosure. An example controller(s) is described below at least with reference to Fig. 6.

[0071] In the example of Fig. 2B, the PiM architecture 200 can represent an example computing system for implementing a neural network or machine-learning model in hardware using a hardware integrated circuit architecture of a memory device. The PiM architecture 200 includes one or more group controllers 204 (“controller 204”) and a host interface 220 that communicates with the SoC 102. As described in more detail below, the host interface 220 and group controllers 204 cooperate to provide datasets and control signals to one or more compute units 212 of the PiM architecture 200.

[0072] The PiM architecture 200 generally includes multiple group controllers 204 that each provide or generate control signals 206 to cause inputs (or activations) to be stored at, or accessed from, memory' locations of shared queue 205. Likewise, the group controllers 204 can also generate control signals 206 to cause weights (or parameters) for a matrix structure of weights (and / or partial sums or accumulated values) to be stored at, or accessed from, memory locations of memory 210 (e.g., DRAM banks).Attorney Docket No. 56113-0798WO1

[0073] The inputs can be represented by an input tensor, whereas the matrix structure of weights is represented by a parameter tensor. Each of the input tensor and the parameter tensor may be multi-dimensional data structures, such as a multi-dimensional matrix or tensor. In some implementations, input samples stored in the shared queue 205 include data values that represent image pixels, word tokens, and / or related activation values, whereas the weights are for a neural network layer through which an input sample is processed to generate a corresponding layer output.

[0074] In some implementations, the shared queue 205 is a hardware memory device. The hardware memory' device can operate as a hardware buffer, a first-in, first-out (FIFO) memory implemented using digital logic (e.g., flip-flops or SRAM) within the PiM architecture 200. The shared queue 205 is managed by pointers (e.g., begin, accumulation, end) and supports read and write operations.

[0075] The PiM architecture 200 includes a shared datapath 207 and multiple computational units (“compute units”) 212. Each compute unit 212 includes arithmetic circuitry with multiply accumulators (MACs) that form one or more MAC cells 214 a / b / c. In some implementations, each compute unit 212 can include N number of cells, where N is an integer greater than or equal to one. The MAC cells 214 perform arithmetic operations such as multiplication and addition. For example, each MAC cell 214 can be used to compute dot products and related matrix multiplications that are performed to convolve an input tensor of image pixel values with a weight tensor (e.g., a kernel filter of weights). In some implementations, the computations are convolution operations performed to generate an output for a convolutional neural network layer.

[0076] In some implementations, the compute units 212 includes an array of MAC cells 214 and PiM blocks 202 of the PiM architecture 200 can be configured to supplement or complement the computational capacity of example special-purpose processor, such as HPU 114 or a neural network processor that executes an ML model to process an inference workload. The PiM architecture 200 leverages compute units 212, timing interval controls, configuration values for input broadcasting (or unicasting) and multi-batch processing to efficiently execute PiM computations by’ exploiting input / output sparsity.

[0077] Leveraging these techniques, for large inference workloads (e.g., LLM or GenAI task), the HPU 114 and / or the SoC 102 can offload a portion of the inference computations to the computational units 212 of PiM architecture 200, thereby expanding or extending the computational capacity’ of the HPU 114. In some implementations, this inference processing can involve computing convolutions, e.g., group convolutions, where data values and / orAttorney Docket No. 56113-0798WO1 operands for an input sample, such as words or pixels, and weight values of a neural network layer are routed to the cells 214 of the compute units 212 to execute the convolution via computations performed at the MAC cells 214 a / b / c. Each weight value can be represented using 2 bytes (2B). More specifically, each weight value can be a 16-bit Bfloat number (BF16) or a 32-bit floating point number (FP32).

[0078] Each group controller 204 can generate control signals 206 to obtain or route operands stored at the shared queue 205 and memory 210 of the PiM architecture 200. For example, group controller 204 can generate control signals 206 to obtain inputs stored at shared queue 205 and weights stored at memory' 210 and initiate / execute neural network and / or machine-learning computations at the compute units 212. Each input obtained from shared queue 205 is routed along the shared datapath 207 to a MAC cell 214 a / b / c in a compute unit 212. The weights obtained from memory 210 can be passed to a corresponding compute unit 212 via a direct datapath based on a tight / close coupling of compute units 212 to DRAM banks / cells of memory' device 122.

[0079] As described below, each compute unit 212 performs computations that produce partial sums or accumulated values for generating outputs for a neural netw ork layer or machine-learning task. In some implementations, an activation function may be applied to a set of outputs to generate a set of output activations for a neural network layer. The outputs (or output activations) computed at the PiM architecture 200 are routed to the shared queue 205 via the shared datapath 207.

[0080] In some implementations, each compute unit 212 generates a set of accumulated values that are reduced to an output or output activation via a reduction operation executed by a reduction circuit of the PiM architecture 200. The reduction circuit is described below at least with reference to the example of Figs. 4-6. In some examples, a set of values associated with a first neural netw ork layer can be generated by a first compute unit 212, routed from the first compute unit 212 to the shared queue 205 via the shared datapath 207, buffered / stored at the shared queue, and routed from the shared queue 205 via the shared datapath 207 to a second, different compute unit 212 for processing at the second compute unit 212 as input activations for a different layer of a neural network implemented using the PiM architecture 200.

[0081] Fig. 3 shows an example parameter tensor 300 with K in N sparsity7, which can represent a uniform sparsity format exhibited by sparse tensors. In general, for K in N sparsity7, for every next N elements along a dimension (e.g.. an innermost dimension) of a parameter tensor, K elements are non-zero.Attorney Docket No. 56113-0798WO1

[0082] One or more opcodes can indicate or specify a sparsity attribute of one or more parameter tensors, as well as sparsity along a particular column (or row) dimension of a given tensor. For example, an opcode(s) in a PiM instruct on / command received at a PiM architecture of memory device 122 can specify a K in N sparsity of a parameter tensor 300, including K in N sparsity of each column 302 or row 304 of the parameter tensor 300. In some implementations, the tensor sparsity information specified by an opcode is based on a structure or configuration of an instruction set used at system 100.

[0083] In the example of Fig. 3, K indicates one or more non-zero values and N is a number of elements associated with the parameter tensor 300. In some examples, N is the number of elements for a given row or column of a parameter tensor. Each of K and N are integers. N can be greater than or equal to one, whereas K can be greater than or equal to zero. The K in N sparsity can be a ratio or some other numerical value that is assigned to, or conveyed as, a sparsity parameter.

[0084] The sparsity parameter characterizes a sparsity attribute or measure of sparsity in a dataset or tensor 300. For example, a sparsity parameter can represent a compression ratio for a given {K. N} pair and is equal to K / N. such that if K=2 and N=4. the compression ratio is 50%. The system 100 can support cases where parameters are compressed with reference to one (or more) dimension(s), such as along a column dimension corresponding to column 302. For this particular type of reduction operation, column 302 can be described as a reduction dimension or an inner product dimension. In some implementations, sparsity in a dataset is based on one or more patterns of sparsity that are detectable during a training phase of a neural network model, a deployment phase of the neural network model, or both.

[0085] The patterns of sparsity can be uniformly distributed among machine-learning datasets, such as parameter tensors 300 that are processed during the training and deployment phases of model execution. The uniformity of the sparsity patterns allows for a certain measure of predictability that can be exploited to realize efficiencies in acceleration of the neural network model implemented using the PiM architecture 200. For example, and as explained below, patterns of sparsity that are uniformly distributed can allow for predicting, inferring, or otherwise detecting an upcoming pattern (e.g.. a sparsity attribute) of zero or non-zero weight values.

[0086] In some implementations, control logic of the PiM architecture is configured to predict, explore, and exploit different sparsity patterns to realize additional efficiencies and optimizations in model execution. In the example of Fig. 3, one or more opcodes received at the PiM architecture 200 can indicate that each of column 302 and row 304 includes a K in NAtorney Docket No. 56113-0798WO1 sparsity of ! , where K=4, N=8. The control logic of the PiM architecture can receive a sparsity’ parameter from the SoC 102 or be configured to determine a value for a sparsity parameter based on the logical expression: % Sparsity = K N.

[0087] In this example a group controller 204 can assign a value of 14 to a respective sparsity' parameter for each of column 302 and row 304. Relatedly, an opcode received at the PiM architecture 200 can also specify that row 306, which may also be a column, includes a K in N sparsity of 5 / 8, where K=5 and N=8. In some implementations, the K for a given K in N sparsity is determined based on a hardware layout of the compute units 212, including the hardware layout of an array of MACs 214 in each compute unit 212. For example, the K can be determined based on a quantity of MAC circuits in a hardware compute cell of a compute unit 212 at a given PiM block.

[0088] Fig. 4 illustrates example configuration options 400, 450 for routing input data / samples to computing resources of the compute units and obtaining reduced outputs from a reduction circuit in a PiM architecture 200. The configuration option 400 is an example input data routing option that can be used to exploit input sparsity, whereas configuration option 450 is an example input data routing option that exploits output sparsity'.

[0089] The configuration options 400 and 450 each include multiple compute units 402, which are indicated as “CU’" in the example of Fig. 4. Each compute unit 402 corresponds to the compute unit 212 described above with reference to the example of Fig. 2B. As described above, each compute unit 402 includes arithmetic circuitry for computing accumulated values from one or more inputs in a set of inputs, which is described in more detail below. In some implementations, the example configuration options 400 and 450 represent data routing options for machine-learning computations related to a layer of a neural network that processes inputs, e.g., one or more arrays of pixel values or word tokens, with a parameter matrix, to generate one or more output values for the neural network layer.

[0090] In configuration options 400 and 450, each of the multiple compute units 402 accepts two values representing operands for a machine-learning (or neural network) computation. A first value / operand 404 is an input that is broadcast to each of the compute units 402. A second value / operand 406 is a parameter (or weight) of a parameter / weight matrix that includes a set of weights 408 for a particular neural network layer. The weights can have vary ing degrees of sparsity as indicated by the different shading / pattems of the example reference blocks that represent different weight values in the example of Fig. 4.

[0091] As described above, to compute an output for neural network layer each value / operand 404 is multiplied with the corresponding second value / operand 406 (e.g.,Attorney Docket No. 56113-0798WO1 corresponding weight) to generate a partial sum, which may be grouped with other partial sums (or accumulated values) to form a set of accumulated values. In some implementations, the weight is passed directly to a multiplier of the compute unit 402, stored in a weight register of the compute unit 402, or both.

[0092] In the example of Fig. 4, the configuration option 400 represents an input broadcast mode, whereas the configuration option 450 represents an output reduction mode. Each mode can be set or defined in a PiM architecture 200 using configuration or control values stored in a mode register of the PiM architecture 200. For the input broadcast mode of configuration option 400, each compute unit 402 multiplies the broadcasted input with a respective weight for the neural network layer and generates a corresponding output value 410, whereas for the output reduction mode of configuration option 450, each of the compute units 402 is coupled to a reduction circuit 412 that receives accumulated values generated by each of the compute units 402 and performs a reduction operation to reduce the accumulated values to a corresponding output value 416. In some examples, the reduction circuit 412 is described alternatively as a reduction tree.

[0093] Fig. 5 illustrates an example configuration 500 that enables data routing and computing resources for exploiting input and output sparsity in a PiM architecture 200. The configuration 500 can operate in a broadcast input mode or an output reduction mode. The broadcast input mode exploits input sparsity, whereas the output reduction mode exploits output sparsity. The PiM architectures described in this document provide architectural solutions that support both input and output sparsity within a single PiM architecture. The configuration 500 includes an input bus / path 502 configured to route input data to each of the compute units 402. In some implementations, the input bus / path 502 coincides with, and / or is a portion of, the shared datapath 207 of the PiM architecture, described above.

[0094] In some implementations, the configuration 500 operates in a broadcast input mode, as described in relation to configuration option 400 of Fig. 4, which includes an input 504 that is broadcast to each of the compute units 402. In some other implementations, the configuration 500 represents an output reduction mode, as described in relation to configuration option 450 of Fig. 4. which includes a distinct input 508 for each compute unit 402. Both modes leverage the shared datapath 207 for data routing within a PiM architecture 200. In some implementations, the shared datapath 207 represents a unified datapath that is shared betw een at least the two distinct modes of a PiM architecture.

[0095] The configuration 500 includes a multiplexer 512 for each compute unit 402. In a broadcast input mode, each multiplexer 512 can be configured to pass the input 504 to one orAttorney Docket No. 56113-0798WO1 more compute units 402, whereas in a unicast input mode each multiplexer 512 can be configured to pass a respective input 508 to a corresponding compute unit 402. For example, during a unicast input mode, one multiplexer 512 can be configured to select and pass input 508-0 to a first compute unit 402 (CU 0), whereas another multiplexer 512 can be configured to select and pass input 508-3 to a second, different compute unit 402 (CU 3). Depending on the active configuration option, e.g., broadcast input mode, unicast input mode, or output reduction mode, the multiplexers 512 routes an appropriate input to the respective compute unit 402. The individual inputs 508 and their corresponding weights 506 (e.g., 506-0, 506-3) can have varying degrees of sparsity as indicated by the different shading / pattems of the example reference blocks that represent different input and weight values in the example of Fig. 5.

[0096] An operative mode of each multiplexer 512 is determined by a control value and implemented by corresponding selection logic of the multiplexer 512. Each multiplexer 512 is configured to route input data provided along the input path 502, which represents a portion of the shared datapath 207. The input data is routed / provided from the shared queue 205, as described below, to the arithmetic circuitry of the respective compute unit 402. and route data from a register file, as described below, to the arithmetic circuitry of the compute unit 402. The shared datapath couples the shared queue to each of the compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture. In some implementations, the operative mode of each multiplexer 512 is determined and set by software instructions based on anticipated sparsity characteristics of data to be processed and output by the compute units 402 and associated components illustrated in configuration 500.

[0097] An output of the selection logic of each multiplexer 512 indicates whether the input 504 is sent to multiple compute units 402 as a first MAC operation value / operand or a respective input 508 is sent to the respective compute unit 402 as the first MAC operation value / operand. Both modes of operation include weight of a parameter / weight matrix (or tensor) as a second value / operand (corresponding weights 506) that is multiplied with a corresponding input (e.g., 504 or 508) to generate a product value associated with a machinelearning or neural network output. A datapath 510 is used to pass a selected weight value (the second value / operand 506) to the compute unit 402.

[0098] In general, the parameter / weight matrix represents a set of weights for a particular neural network layer. In some examples, a respective value of each weight in a subset of weights in the parameter / weight matrix is zero, which indicates a measure of sparsity’ in theAttorney Docket No. 56113-0798WO1 weight matrix. Relatedly, a respective value of each weight in another subset of weights in the parameter / weight matrix is non-zero, which indicates a set of non-sparse weight values in the weight matrix. In some cases, the weights exhibit sparsity as features (e.g., rows of the matrix) that include zeros, or near zero, values, which can result in a sparse output when the matrix is multiplied with respective input values.

[0099] The configuration 500 includes functionality of configuration option 400 and of configuration option 450, in which the control value determines a setting of the multiplexers 512 that determines which configuration option (e.g., 400 or 450) is activated. Depending on the active mode of configuration 500 (determined by the control parameter), the outputs of each compute unit 402 are coupled to, routed along, one of two output paths. As described in relation to Fig. 4, for the input broadcast mode of configuration 500, each compute unit 402 multiplies the broadcasted input with a respective weight for the neural network layer and generates a corresponding output value 410.

[0100] For example, each of output values 410-0, 410-1, 410-2, and 410-3 is propagated along output datapath 516. whereas during the output reduction mode of configuration 500, each of the compute units 402 (e.g., CU 0. CU 1, CU 2. CU 3) is coupled to a reduction circuit 412 that receives accumulated values via datapath 514. In the output reduction mode, the output values generated by each of the compute units 402 represent accumulated values and the reduction circuit 412 performs a reduction operation on the accumulated values to reduce the accumulated values to a corresponding output value 416.

[0101] The reduction circuit 412 includes a first portion of circuitry (or first circuit) that is a hardware reduction block that includes an adder tree configured to perform accumulation operations on partial sums generated by one or more of the compute units 402. The adder tree of the first portion of circuitry is configured to receive multiple inputs and to generate a single output, the output value 416. The reduction circuit 412 also includes a second portion of circuitry (or second circuit) that includes a hardware adder tree for each set of compute units 402 that are disposed adjacent, and coupled to, a corresponding bank group of memory 210. In addition, each respective hardware adder tree is configured to perform accumulation operations on partial sums generated by the set of compute units 402 to which the respective hardware adder is coupled. Each of the adder trees of the second portion of circuity is configured to receive multiple inputs from a single compute unit 402 and to generate a single output, which is in turn passed to the adder tree of the first portion of circuitry'. This is shown at Fig. 6 and described in more detail with reference to the example of Fig. 6.Attorney Docket No. 56113-0798WO1

[0102] Fig. 6 illustrates an example PiM architecture 200 that includes a shared datapath and shared queue for exploiting both input and output sparsity. Fig. 6 illustrates datapaths for an input broadcast mode configuration 600 of the PiM architecture and datapaths for an output reduction mode configuration 650 of PiM architecture. The physical layout of components and datapaths for both modes is the same, but one or more components, e.g., multiplexers, are configured differently depending on the active mode of the architecture, where a control value determines the active mode. The example configuration 500 is illustrated in the center of Fig. 6 and provides a schematic illustration to demonstrate the operation of the architectures illustrated in the input broadcast mode configuration 600 and the output reduction mode configuration 650.

[0103] The dotted lines in configuration 600 illustrate a datapath for the input broadcasting mode. The data flows from an I / O interface 220, to a shared queue 612, to the compute units 402 across all DRAM banks 604. The dotted lines in configuration 650 represent a datapath for the output reduction mode. The data flows from each compute unit 402 to the shared queue 612 and to the I / O interface 220.

[0104] Both configurations 600 and 650 include multiple DRAM banks 604. where each DRAM bank 604 includes multiple DRAM memory devices, e.g., DRAM block 605. In some implementations, the DRAM blocks store weights of a corresponding neural network layer, and / or values of an input activation processed by the neural network layer, and / or output values of the neural network layer.

[0105] In addition to the DRAM banks, the configuration 600 and 650 include a bank group controller 204 (not illustrated in Fig. 6, but illustrated in Fig. 2. The bank group controller 204 is communicatively coupled to the DRAM banks 604, the compute units 402, and shared queue 612), the I / O interface 220, multiple compute units 402, as described in relation to Fig. 4, reduction tree circuitry 610, and the shared queue 612. Some configurations do not use one or more components of the configuration 600 and 650, depending on the particular configuration. For example, the reduction tree circuitry 610 is not active when the configuration 600 is implemented. The shared queue 612 holds the input broadcast values, as described in relation to Fig. 4 while operating with the configuration 600, and output values of the layer operation while operating with the configuration 650.

[0106] The bank group controller 204 implements control logic for generating control signals that cause components of the configuration 600 and 650 to perform one or more functions. The bank group controller 204 is configured to generate control signaling used to route one or more inputs to particular compute units. In some implementations, configurationAttorney Docket No. 56113-0798WO1600 and 650 are subsystems of system 100 and represents an improved PiM architecture that enhances performance and energy efficiency by exploiting input and output sparsity during matrix x matrix operations for machine-learning computations performed in a DRAM device, such as memory device 122.

[0107] The I / O host interface 220 facilitates storing of input values from a host device to a shared queue 612, in which the shared queue 612 can store input activation values, as well as output values. For example, partial sums of reduction operations can be stored in the shared queue 612 for the output reduction mode PiM configuration 650. Input activation values are distributed to the multiple compute units 402 via the shared datapaths. The I / O host interface 220 is for storing input samples (e.g., pixels, words, or activation values), and the DRAM banks, e.g., the DRAM bank 604 is for storing sets of weights for a neural network layer through which an input sample is processed to generate a corresponding layer output.

[0108] As implemented in configuration 650, the reduction tree circuitry 610 reduces output from multiple compute units 402 into a single output. In some implementations, the reduction tree circuitry 610 includes multiple reduction steps, associated with multiple partial sums. The output reduction mode PiM configuration 650 includes the shared queue 612 that receives output values from the reduction tree circuitry 610. The reduction tree circuitry 610 processes outputs from the multiple compute units 402, which execute MAC operations between weights, stored in DRAM blocks 605 and the input activation values.

[0109] Fig. 7 illustrates an input bus that is shared between two modes of operation and example features of a computing unit in a PiM architecture. Fig. 7 illustrates components of the example configuration 500 that exploit both input and output sparsity in a single shared architecture and the input broadcast mode PiM configuration 600. In addition, further detail of compute unit components 700 of the compute unit 402 is illustrated.

[0110] Each compute unit 402 includes selection logic to distribute inputs for both input broadcasting and output reduction modes. The compute unit components 700 include MAC units 702, a selection bit input 704, multiplexers 706, and a PiM register file 708. The MAC units 702 of each compute unit 402 perform MAC operations associated with a neural network layer. The MAC units 702 receive operands from a respective multiplexer 706. For the input broadcast mode PiM configuration 600, the operands are a broadcasted input value and a weight value. For the output reduction mode PiM configuration 650, the operands are a respective input value and a respective weight value. The selection bit input 704 determines a configuration of each multiplexer 706, where the value of the selection bit input 704Attorney Docket No. 56113-0798WO1 determines the mode of the architecture. For example, in some cases a bit value of 0 can configure the architecture to operate in configuration 600 and a bit value of 1 can configure the architecture to operate in configuration 650.

[0111] The selection logic, e.g., logic implemented by the multiplexers 706, enables adaptive input handling based on the sparsity-exploitation mode. In other words, the sy stem can either broadcast data from the shared queue or unicast data from a data buffer within the PiM register file 708. By sharing an input datapath for both modes, the architecture minimizes area overhead. The MAC units 702 and the PiM register file 708 are shared between the two sparsity-exploitation modes.

[0112] The register file 708 is configured to buffer input data for a given compute operation, and the selection logic is implemented by the multiplexer 706 with hardware configured to route data provided along the shared datapath from a shared queue, e.g., the shared queue 612, to the arithmetic circuitry of the compute unit 402. In addition, the hardware is configured to route data from the register file 708 to the arithmetic circuitry of the compute units 402. In both configuration modes, e.g., the input broadcast mode and the output reduction mode, each respective selection logic hardware of each compute unit 402 is configured to distribute the one or more inputs to the compute unit.

[0113] In some implementations, each hardware multiplexer 706 of each compute unit is configured to enable adaptive input handling at the PiM architecture based on a sparsityexploitation mode of the PiM architecture. In some implementations, the adaptive input handling includes broadcasting inputs routed from the shared queue and unicasting input data from a data buffer of the register file 708 of the corresponding compute unit.

[0114] In some implementations, each multiplexer 706 implements selection logic in a respective compute unit, which is configured to receive a select bit for its hardware multiplexer, select a particular input data source based on the select bit, and route input data to the arithmetic circuitry of the respective compute unit based on the particular input data source that is selected based on the select bit.

[0115] In some implementations, each multiplexer 706 implements selection logic in a respective compute unit, which is configured to select input data buffered in its register file as the particular input data source based on a first value of the select bit and route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.

[0116] In some implementations, each multiplexer 706 implements selection logic in a respective compute unit, which is configured to select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit, and route,Attorney Docket No. 56113-0798WO1 from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

[0117] Fig. 8 illustrates an example configuration of a PiM architecture 800 that leverages the shared datapath of at least Fig. 7 to enable element-wise operations for executing machine-learning computations. As described in relation to Fig. 6, the configuration 600 illustrates an input broadcasting mode, where the data flows from an I / O interface to a shared queue, and then to each of the PiM compute units. In this mode, the reduction tree circuitry as illustrated in configuration 600 is disabled.

[0118] In addition to enabling input sparsity' exploitation, the configuration 600 enables element-wise operations without adding extra datapaths or logic to execute the operations between data stored in the DRAM banks and data stored in the PiM register, or between two data elements stored in the PiM register. By reusing the datapath, e.g., the input path from input reduction mode and the output path from the input-broadcasting mode, element-wise operations are naturally enabled.

[0119] Fig. 9 illustrates an example schematic 900 input / output (I / O) queue that is shared between two modes of operation in a PiM architecture. The logic illustrated by schematic 900 describes queue operation for writing and / or accumulating outputs when the PiM architecture operates in an output reduction mode, as described in relation to configuration 650 of Fig. 6.

[0120] In some implementations, the reduction tree circuitry 902 receives outputs from multiple compute units, e.g.. the compute units 402. The outputs correspond to partial sums that are reduced by the reduction tree circuitry 902 to obtain a single output value.

[0121] The schematic 900 includes a PiM I / O queue 904. To minimize the size of partial sums transferred to a host device, the PiM I / O queue 904 is used to accumulate intra-rank reduction results directly within the PiM device. This is particularly beneficial when multiple Zin blocks (groups of weights in a DRAM column granularity sharing Zin indices) are assigned per PiM compute unit.

[0122] Two registers facilitate the intra-rank accumulation. First, a PiM I / O queue accumulation pointer points to a location in the PiM I / O queue 904 where new incoming partial sums through a write path from the reduction tree circuitry 902 are accumulated. Second, an intra-rank reduction accumulation count register determines how many accumulations occur in a same I / O queue location pointed by the PiM I / O queue accumulation pointer before the accumulation pointer advances.

[0123] The accumulation process begins by initializing all PiM I / O queues and associated pointers to zero, e.g., begin, end, and accumulation pointers. The end pointer is incrementedAttorney Docket No. 56113-0798WO1 before an initial partial sum write, which occurs when the end and accumulation pointers align. The accumulation pointer increments independently, advancing only after a number of accumulations at its current location reaches a limit specified in the intra-rank reduction accumulation count. The accumulation process continues, even if the end pointer reaches the end of the queue. In such cases, the accumulation pointer wraps around and continues from the beginning of the queue.

[0124] Fig. 10 is an example process for exploiting input and output sparsity for inmemory machine-learning computations executed using a PiM architecture.

[0125] Process 1000 is implemented or executed at system 100 using at least the SoC 102 and memory device 122 described above, including the integrated systems, circuits, and resources of the different configurations of the PiM architecture described herein. Hence, descriptions of process 1000 will reference at least the above-mentioned data routing and computing resources of memory device 122. In some examples, the steps or actions of process 1000 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transi toiy machine-readable storage device and is executable by one or more of the processors, processing devices, or other resources described in this specification.

[0126] Referring again to process 1000, the system 100 stores (1002) a set of inputs in a shared queue of the PiM architecture. In some implementations, the set of inputs are input activation values for performing computations associated with a layer of a neural network. In some cases, the input actuation values are an output from a previous neural network layer. In some other cases, the input actuation values are an initial set of inputs, e.g., an image from a camera or sensor, that is processed by layers of a neural network.

[0127] The system 100 provides (1004). by way of a shared datapath of the PiM architecture, one or more inputs in the set of inputs from the shared queue to multiple compute units in the PiM architecture. In some implementations, each compute unit performs one or more MAC operations between one or more inputs in the set of inputs and corresponding weights, where the weights are associated with a particular layer of the neural network. The multiple compute units of system 100 compute (1006) accumulated values from the one or more inputs.

[0128] In some implementations, by way of the shared datapath, the system 100 provides (1008) the accumulated values to reduction circuitry in the PiM architecture configured to perform reduction operations on the accumulated values. In the case of the output reduction mode, as described in relation to configuration 650, the reduction circuitry reduces theAttorney Docket No. 56113-0798WO1 outputs from each compute unit into a single output value. In the case of the input broadcast mode, as described in relation to configuration 600, each compute unit generates an independent output and the reduction circuitry is disabled.

[0129] In some implementations, the system 100 generates (1010) a machine-learning output from the accumulated values based on the reduction operations. Each accumulated value received from a compute unit corresponds to a partial sum of the reduced output.

[0130] The system stores (1012) the machine learning output in the shared queue in response to routing the machine-learning output to the shared queue using the shared datapath.

[0131] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[0132] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0133] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g.. code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0134] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages,Attorney Docket No. 56113-0798WO1 or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0135] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication netw ork.

[0136] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0137] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.

[0138] Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g.. a universal serial bus (USB) flash drive, to name just a few.

[0139] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memorw media and memory devices, including by w ay of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks;Attorney Docket No. 56113-0798WO1 and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0140] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid cry stal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0141] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0142] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0143] In addition to the embodiments described above, the following embodiments are also innovative:

[0144] Embodiment 1 is a Processing-in-Memory (“PiM”) architecture for a memory' device, comprising:

[0145] a shared queue configured to store a set of inputs and one or more outputs;Attorney Docket No. 56113-0798WO1

[0146] a plurality of compute units, each of the plurality of compute units comprising arithmetic circuitry for computing accumulated values from one or more inputs in the set of inputs;

[0147] reduction circuitry configured to generate the one or more outputs based on reduction operations performed on the accumulated values; and

[0148] a shared datapath configured to (i) provide the one or more inputs to the plurality of compute units, (ii) provide the accumulated values to the reduction circuitry; and (iii) provide the one or more outputs to the shared queue.

[0149] Embodiment 2 is the architecture of embodiment 1 , wherein the shared datapath couples the shared queue to each of the plurality of compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture.

[0150] Embodiment 3 is the architecture of any of embodiments 1 -2, wherein the shared datapath is configured as a unified datapath that is shared between at least two distinct modes of the PiM architecture.

[0151] Embodiment 4 is the architecture of embodiment 3, wherein the at least two distinct modes of the PiM architecture comprises (i) an input broadcast mode where one or more inputs are broadcast to each of the plurality of compute units and (ii) a reduction mode where one or more outputs are generated based on the reduction operations.

[0152] Embodiment 5 is the architecture of embodiment 4, further comprising:

[0153] an input unicast mode where one or more inputs are routed to a particular compute unit of the plurality' of compute units.

[0154] Embodiment 6 is the architecture of any of embodiments 1-5, wherein each of the plurality of compute units comprises:

[0155] a register file configured to buffer input data for a given compute operation; and

[0156] selection logic comprising a hardware multiplexer configured to:

[0157] (i) route data provided along the shared datapath from the shared queue to the arithmetic circuitry of the compute unit, and

[0158] (ii) route data from the register file to the arithmetic circuitry of the compute unit.

[0159] Embodiment 7 is the architecture of embodiment 6, wherein each respective selection logic in a compute unit of the plurality' of compute units is configured to distribute the one or more inputs to the compute unit during each of an input broadcast mode of the PiM architecture and an input unicast mode of the PiM architecture.Attorney Docket No. 56113-0798WO1

[0160] Embodiment 8 is the architecture of embodiment 7, wherein each hardware multiplexer in each respective selection logic is configured to enable adaptive input handling at the PiM architecture based on a sparsity-exploitation mode of the PiM architecture.

[0161] Embodiment 9 is the architecture of embodiment 6, wherein the adaptive input handling comprises:

[0162] (i) broadcasting inputs routed from the shared queue; and

[0163] (ii) unicasting input data from a data buffer of the register file of the corresponding compute unit.

[0164] Embodiment 10 is the architecture of any one of embodiments 6 to 9, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:

[0165] (i) receive a select bit for its hardware multiplexer;

[0166] (ii) select, by its hardware multiplexer, a particular input data source based on the select bit; and

[0167] (iii) route input data to the arithmetic circuitry of the compute unit based on the particular input data source that is selected based on select bit.

[0168] Embodiment 11 is the architecture of embodiment 10, wherein each respective selection logic in a compute unit of the plurality' of compute units is configured to:

[0169] (i) select input data buffered in its register file as the particular input data source based on a first value of the select bit; and

[0170] (ii) route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.

[0171] Embodiment 12 is the architecture of embodiment 10 or 11, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:

[0172] (i) select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit; and

[0173] (ii) route, from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

[0174] Embodiment 13 is the architecture of any of embodiments 1-12. wherein the reduction circuitry comprises:

[0175] a first circuit that is a hardware reduction block comprising an adder tree configured to perform accumulation operations on partial sums generated by one or more of the plurality of compute units.Attorney Docket No. 56113-0798WO1

[0176] Embodiment 14 is the architecture of embodiment 13, wherein the reduction circuitry comprises:

[0177] a second circuit comprising a respective hardware adder tree for each set of compute units that are disposed adjacent, and coupled to, a corresponding bank group of the memory device; and

[0178] each respective hardware adder tree of the second circuity is configured to perform accumulation operations on partial sums generated by the set of compute units to which the respective hardware adder is coupled.

[0179] Embodiment 15 is the architecture of any of embodiments 1-14, further comprising:

[0180] a bank group controller configured to generate control signaling used to route one or more inputs to particular compute units of the plurality of compute units.

[0181] Embodiment 16 is a method performed using a Processing-in-Memory (“PiM”) architecture of an integrated memory device, the method comprising:

[0182] storing a set of inputs in a shared queue of the PiM architecture;

[0183] providing, by way of a shared datapath of the PiM architecture, one or more inputs in the set of inputs from the shared queue to a plurality of compute units in the PiM architecture;

[0184] computing, by the plurality of compute units, accumulated values from the one or more inputs;

[0185] providing, via the shared datapath, the accumulated values to reduction circuitry in the PiM architecture configured to perform reduction operations on the accumulated values;

[0186] generating a machine-learning output from the accumulated values based on the reduction operations; and

[0187] storing the machine-learning output in the shared queue in response to routing the machine-learning output to the shared queue using the shared datapath.

[0188] Embodiment 17 is the method of embodiment 16, wherein the shared datapath couples the shared queue to each of the plurality of compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture.

[0189] Embodiment 18 is the method of any of embodiments 16-17, wherein the shared datapath is configured as a unified datapath that is shared between at least two distinct modes of the PiM architecture.Attorney Docket No. 56113-0798WO1

[0190] Embodiment 19 is the method of embodiment 18, wherein the at least two distinct modes of the PiM architecture comprises:

[0191] (i) an input broadcast mode where one or more inputs are broadcast to each of the plurality of compute units; and

[0192] (ii) a reduction mode where one or more outputs are generated based on the reduction operations.

[0193] Embodiment 20 is the method of embodiment 19, further comprising:

[0194] an input unicast mode where one or more inputs are routed to a particular compute unit of the plurality of compute units.

[0195] Embodiment 21 is the method of any of embodiments 16-20, further comprising:

[0196] buffering, by a register file of each compute unit of the plurality of compute units, input data for a given compute operation;

[0197] routing, by a hardware multiplexer of each compute unit configured to perform a respective selection logic, data provided along the shared datapath from the shared queue to the arithmetic circuitry of the compute unit; and

[0198] routing, by the hardware multiplexer, data from the register file to the arithmetic circuitry of the compute unit.

[0199] Embodiment 22 is the method of embodiment 21, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to distribute the one or more inputs to the compute unit during each of an input broadcast mode of the PiM architecture and an input unicast mode of the PiM architecture.

[0200] Embodiment 23 is the method of any of embodiments 16-22, wherein each hardware multiplexer in each respective selection logic is configured to enable adaptive input handling at the PiM architecture based on a sparsity-exploitation mode of the PiM architecture.

[0201] Embodiment 24 is the method of embodiment 23, wherein the adaptive input handling comprises:

[0202] (i) broadcasting inputs routed from the shared queue; and

[0203] (ii) unicasting input data from a data buffer of the register file of the corresponding compute unit.

[0204] Embodiment 25 is the method of any one of embodiments 21 to 24, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:

[0205] (i) receive a select bit for its hardware multiplexer;Attorney Docket No. 56113-0798WO1

[0206] (ii) select, by its hardware multiplexer, a particular input data source based on the select bit; and

[0207] (iii) route input data to the arithmetic circuitry of the compute unit based on the particular input data source that is selected based on select bit.

[0208] Embodiment 26 is the method of embodiment 25, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:

[0209] (i) select input data buffered in its register file as the particular input data source based on a first value of the select bit; and

[0210] (ii) route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.

[0211] Embodiment 27 is the method of embodiment 25 or 26, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:

[0212] (i) select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit; and

[0213] (ii) route, from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

[0214] Embodiment 28 is the method of any of embodiments 16-27, wherein the reduction circuitry7comprises:

[0215] a first circuit that is a hardware reduction block comprising an adder tree configured to perform accumulation operations on partial sums generated by one or more of the plurality of compute units.

[0216] Embodiment 29 is the method of embodiment 28, wherein the reduction circuitry7comprises:

[0217] a second circuit comprising a respective hardware adder tree for each set of compute units that are disposed adjacent, and coupled to, a corresponding bank group of the memory device; and

[0218] each respective hardware adder tree of the second circuit is configured to perform accumulation operations on partial sums generated by the set of compute units to which the respective hardware adder is coupled.

[0219] Embodiment 30 is the method of any of embodiments 16-29, further comprising:

[0220] generating, by a bank controller, control signaling used to route one or more inputs to particular compute units of the plurality of compute units.

[0221] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may beAttorney Docket No. 56113-0798WO1 claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0222] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0223] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

Atorney Docket No. 56113-0798WO1What is claimed is:

1. A Processing-in-Memory ( ‘PiM”) architecture for a memory device, comprising: a shared queue configured to store a set of inputs and one or more outputs; a plurality' of compute units, each of the plurality' of compute units comprising arithmetic circuitry for computing accumulated values from one or more inputs in the set of inputs; reduction circuitry configured to generate the one or more outputs based on reduction operations performed on the accumulated values; and a shared datapath configured to:(i) provide the one or more inputs to the plurality of compute units,(ii) provide the accumulated values to the reduction circuitry; and(iii) provide the one or more outputs to the shared queue.

2. The PiM architecture of claim 1, wherein the shared datapath couples the shared queue to each of the plurality of compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture.

3. The PiM architecture of any of claims 1-2. wherein the shared datapath is configured as a unified datapath that is shared between at least two distinct modes of the PiM architecture.

4. The PiM architecture of claim 3, wherein the at least two distinct modes of the PiM architecture comprises:(i) an input broadcast mode yvhere one or more inputs are broadcast to each of the plurality of compute units; and(ii) a reduction mode where one or more outputs are generated based on the reduction operations.

5. The PiM architecture of claim 4, further comprising: an input unicast mode yvhere one or more inputs are routed to a particular compute unit of the plurality of compute units.Attorney Docket No. 56113-0798WO16. The PiM architecture of any of claims 1-5. wherein each of the plurality of compute units comprises: a register file configured to buffer input data for a given compute operation; and selection logic comprising a hardware multiplexer configured to:(i) route data provided along the shared datapath from the shared queue to the arithmetic circuitry of the compute unit, and(ii) route data from the register file to the arithmetic circuitry of the compute unit.

7. The PiM architecture of claim 6, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to distribute the one or more inputs to the compute unit during each of an input broadcast mode of the PiM architecture and an input unicast mode of the PiM architecture.

8. The PiM architecture of claim 7, wherein each hardware multiplexer in each respective selection logic is configured to enable adaptive input handling at the PiM architecture based on a sparsity-exploitation mode of the PiM architecture.

9. The PiM architecture of claim 8, wherein the adaptive input handling comprises:(i) broadcasting inputs routed from the shared queue; and(ii) unicasting input data from a data buffer of the register file of the corresponding compute unit.

10. The PiM architecture of any one of claims 6 to 9, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:(i) receive a select bit for its hardware multiplexer;(ii) select, by its hardware multiplexer, a particular input data source based on the select bit; and(iii) route input data to the arithmetic circuitry of the compute unit based on the particular input data source that is selected based on select bit.

11. The PiM architecture of claim 10, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:Attorney Docket No. 56113-0798WO1(i) select input data buffered in its register file as the particular input data source based on a first value of the select bit; and(ii) route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.

12. The PiM architecture of claim 10 or 11, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:(i) select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit; and(ii) route, from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

13. The PiM architecture of any of claims 1-12, wherein the reduction circuitry comprises: a first circuit that is a hardware reduction block comprising an adder tree configured to perform accumulation operations on partial sums generated by one or more of the plurality of compute units.

14. The PiM architecture of claim 13, wherein the reduction circuitry comprises: a second circuit comprising a respective hardware adder tree for each set of compute units that are disposed adjacent, and coupled to, a corresponding bank group of the memory device; and each respective hardware adder tree of the second circuit is configured to perform accumulation operations on partial sums generated by the set of compute units to which the respective hardware adder is coupled.

15. The PiM architecture of any of claims 1-14, further comprising: a bank group controller configured to generate control signaling used to route one or more inputs to particular compute units of the plurality of compute units.

16. A method performed using a Processing-in-Memory ("PiM") architecture of an integrated memory device, the method comprising: storing a set of inputs in a shared queue of the PiM architecture;Attorney Docket No. 56113-0798WO1 providing, by way of a shared datapath of the PiM architecture, one or more inputs in the set of inputs from the shared queue to a plurality of compute units in the PiM architecture; computing, by the plurality of compute units, accumulated values from the one or more inputs; providing, via the shared datapath, the accumulated values to reduction circuitry7in the PiM architecture configured to perform reduction operations on the accumulated values; generating a machine-learning output from the accumulated values based on the reduction operations; and storing the machine-learning output in the shared queue in response to routing the machine-learning output to the shared queue using the shared datapath.

17. The method of claim 16, wherein the shared datapath couples the shared queue to each of the plurality of compute units and is configured to route inputs and outputs along the shared datapath based on a particular mode of the PiM architecture.

18. The method of any of claims 16-17, wherein the shared datapath is configured as a unified datapath that is shared between at least two distinct modes of the PiM architecture.

19. The method of claim 18, wherein the at least two distinct modes of the PiM architecture comprises:(i) an input broadcast mode where one or more inputs are broadcast to each of the plurality of compute units; and(ii) a reduction mode where one or more outputs are generated based on the reduction operations.

20. The method of claim 19, further comprising: an input unicast mode where one or more inputs are routed to a particular compute unit of the plurality of compute units.

21. The method of any of claims 16-20, further comprising: buffering, by a register file of each compute unit of the plurality of compute units, input data for a given compute operation;Attorney Docket No. 56113-0798WO1 routing, by a hardware multiplexer of each compute unit configured to perform a respective selection logic, data provided along the shared datapath from the shared queue to arithmetic circuitry of the compute unit; and routing, by the hardware multiplexer, data from the register file to the arithmetic circuitry7of the compute unit.

22. The method of claim 21. wherein each respective selection logic in a compute unit of the plurality of compute units is configured to distribute the one or more inputs to the compute unit during each of an input broadcast mode of the PiM architecture and an input unicast mode of the PiM architecture.

23. The method of claim 22, wherein each hardware multiplexer in each respective selection logic is configured to enable adaptive input handling at the PiM architecture based on a sparsity-exploitation mode of the PiM architecture.

24. The method of claim 23. wherein the adaptive input handling comprises:(i) broadcasting inputs routed from the shared queue; and(ii) unicasting input data from a data buffer of the register file of the corresponding compute unit.

25. The method of any one of claims 21 to 24, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:(i) receive a select bit for its hardware multiplexer;(ii) select, by its hardware multiplexer, a particular input data source based on the select bit; and(iii) route input data to the arithmetic circuitry of the compute unit based on the particular input data source that is selected based on select bit.

26. The method of claim 25. wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:(i) select input data buffered in its register file as the particular input data source based on a first value of the select bit; and(ii) route the selected input data buffered in its register file to the arithmetic circuitry of the compute unit.Atorney Docket No. 56113-0798WO1 l ' l . The method of claim 25 or 26, wherein each respective selection logic in a compute unit of the plurality of compute units is configured to:(ii) select the set of inputs stored in the shared queue as the particular input data source based on a second value of the select bit; and(ii) route, from the shared datapath, the selected set of inputs stored in the shared queue to the arithmetic circuitry of the compute unit.

28. The method of any of claims 16-27, wherein the reduction circuitry comprises: a first circuit that is a hardware reduction block comprising an adder tree configured to perform accumulation operations on partial sums generated by one or more of the plurality of compute units.

29. The method of claim 28, wherein the reduction circuitry comprises: a second circuit comprising a respective hardware adder tree for each set of compute units that are disposed adjacent, and coupled to. a corresponding bank group of the memory device; and each respective hardware adder tree of the second circuit is configured to perform accumulation operations on partial sums generated by the set of compute units to which the respective hardware adder is coupled.

30. The method of any of claims 16-29, further comprising: generating, by a bank controller, control signaling used to route one or more inputs to particular compute units of the plurality of compute units.

Images