Multi-batch computations in a processing-in-memory architecture of a memory circuit

Abstract

Methods, systems, and apparatus, including computational instructions / programs encoded on non-transitory computer-readable media, are disclosed for executing multi-batch computations using a memory circuit having a processing-in-memory ("PiM") block. During a first timing interval, a system that includes the memory circuit: i) stores a first weight value and a first input in a first compute cell of the PiM block; and ii) performs a first computation at the PiM block using the first compute cell, the first weight value, and the first input. During a second timing interval that overlaps with the first timing interval, the system: i) stores a second, different input in a second compute cell of the PiM block; and ii) performs a second computation at the PiM block using the second compute cell, the first weight value, and the second input.

Description

MULTI-BATCH COMPUTATIONS IN A PROCESSING-IN-MEMORY ARCHITECTURE OF A MEMORY CIRCUITBACKGROUND

[0001] This specification generally relates to memory devices used to execute computations.

[0002] Modem computing systems often incorporate a wide variety of compute processing units that each offer different computing capabilities and trade-offs. Efficient execution of a given compute job often involves parsing computations into meaningful subtasks or workloads that are mapped to available processor cores of a computing system. The computations may be parsed and mapped based on suitability criteria, such as processor capability', performance, and power. Generally, this overall process of allocating portions of a computation to appropriate processor resources is referred to as heterogeneous compute.

[0003] At least one processor core of the computing system can be an Intellectual Property block (“IP block") that executes a respective portion of a computational operation for different multimedia workloads. Example use cases can involve processing image or speech data captured respectively by a camera or microphone on the mobile device as well as performing computations for generative artificial intelligence (“GenAT) applications. The system-on-chip can use a heterogeneous compute operation to process input samples derived from image data, speech data, a text corpus, or a combination of these. An example step in the heterogeneous compute operation can include processing data associated with the input samples using a memory device that provides in-memory processing or computing capabilities.SUMMARY

[0004] This specification describes hardware and software techniques for accelerated multi-batch processing in a memory device coupled to a system-on-chip (“SoC”). The memory device can be a dynamic random-access memory (DRAM) device that includes a Processing-in-Memory (“PiM”) architecture configured to perform memory -bounded and compute-bounded computations for executing an inference workload or task in the memory device. The PiM architecture defines one or more PiM blocks of a memory device and each PiM block includes compute elements, such as a processor unit, mode registers, and one or more computational units, e.g., arithmetic logic units (ALUs) or related addition and multiplication circuitry. For example, the PiM block can include discrete processors,processor units, register devices, buffers, multiply accumulate cells (MACs), etc. that cooperate to form one or more PiM compute elements.

[0005] The PiM blocks are used to execute computations for an example workload, such as a machine-learning workload for computing outputs associated with a generative artificial intelligence (“GenAI”) application. The computations can be segmented into respective portions that are allocated between the SoC and one or more PiM blocks of the memory device. The disclosed techniques for executing multi-batch computations in the memory device include implementing unique and overlapping data access (e.g., read) intervals and computation intervals using resources of the PiM blocks. The computation intervals include at least two distinct compute pipeline intervals, which are staggered to enable concurrent processing of respective batches of inputs.

[0006] For example, a first compute pipeline interval (“compute interval”) of the PiM block is used to process machine-learning inputs of a first batch of inputs and a second, different compute pipeline interval of the PiM block is used to process machine-learning inputs of a second batch of inputs. Each of the first and second compute intervals represent different pipeline stages of the PiM block and are implemented based on a respective subset of overlapping clock cycles. Timing aspects of the pipeline stages can be staggered to efficiently support multi-batch computations. For example, the compute intervals for the different pipeline stages can be staggered using at least a one clock cycle offset to support data (e.g., weight value) reuse in a manner that enables efficient, concurrent processing of inputs across multiple input batches.

[0007] The techniques include generating data and control signaling at the SoC, which are used to communicate with the PiM architecture and memory' device by way of a memory controller of the SoC. For example, the data and control signaling generated at the SoC are passed to, and processed by, compute elements of the PiM blocks within a given PiM architecture to execute the computations for an example workload. In some implementations, the data / control signaling are processed at the PiM architecture to trigger execution of certain data processing and computing operations using the compute intervals that represent the different pipeline stages of the PiM block. The control signals can also be generated locally at the memory device, which may be external to, or separate from, the SoC.

[0008] One aspect of the subject matter described in this specification can be embodied in a multi-batch data processing method implemented using an integrated circuit comprising an input register, a PiM block, and a storage bank. The method includes, during a first timing interval, storing, in a compute cell of the PiM block, a first weight value obtained from thestorage bank and a first input obtained from the input register; and performing a first computation at the PiM block using the compute cell, the first weight value, and the first input. The method further includes, during a second timing interval that overlaps with the first timing interval, storing, in the compute cell of the PiM block, a second, different input obtained from the input register; and performing a second computation at the PiM block using the compute cell, the first weight value, and the second input.

[0009] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, the method further includes generating control signals using a control logic of the integrated circuit; initiating a multibatch compute mode of the PiM block based on the control signals; and performing the first and second computations during the multi-batch compute mode of the PiM block. The method further includes, in response to initiating the multi-batch compute mode of the PiM block, concurrently processing inputs of at least two different batches of inputs by reusing the first weight value, such that processing of the first input overlaps with processing of the second input.

[0010] In some implementations, performing the second computation includes performing the second computation during a portion of the second timing interval that overlaps with the first timing interval. Performing the second computation can include performing the second computation concurrently with performing the first computation. In some implementations, the first input obtained from the input register is from a first batch of inputs; and the second, different input obtained from the input register is from a second, different batch of inputs. The first timing interval can be a first compute pipeline interv al of the PiM block, for the first batch of inputs; and the second timing interval can be a second compute pipeline interval of the PiM block, for the second batch of inputs.

[0011] In some implementations, performing the first computation includes: i) performing, at the compute cell and during the first compute pipeline interval, a first multiplication operation to multiply the first input and the first weight value; ii) generating a first product based on the first multiplication operation; and iii) performing a first addition operation to accumulate the first product and a different product. In some implementations, performing the second computation comprises: i) performing, at the compute cell and during the second compute pipeline interval, a second multiplication operation to multiply the second input and the first weight value; ii) generating a second product based on the second multiplication operation; and iii) performing a second addition operation to accumulate the second product and a different product.

[0012] The first addition operation and the second multiplication operation can be performed concurrently based on a clock cycle of the first timing interval that overlaps with a clock cycle of the second timing interval. In some implementations, the method further includes, during a first read interval of the PiM block, i) obtaining the first weight value from the storage bank; and ii) obtaining the first input from the input register; and during a second read interval of the PiM block: i) obtaining the second input from the input register.

[0013] Each read interval of the PiM block can execute read operations based on a TCCD S timing constraint. In some implementations, each read interval of the PiM block operates based on a first frequency, / !; and each compute pipeline interval of the PiM block operates based on a second frequency, fl, that is different from the first frequency, / !. In some implementations, the / 2 is at least ! of / !. The method further includes performing computations for multiple inputs using multiple pipeline stages of the PiM block. Each input of the multiple inputs corresponds to a distinct batch and each batch corresponds to a distinct pipeline stage of the multiple pipeline stages.

[0014] Another aspect of the subject matter described in this specification can be embodied in a system comprising a processing device; an integrated circuit comprising an input register, a processing-in-memory (“PiM’’) block, and a storage bank; and a machine- readable storage device for storing instructions that are executable by the processing device to cause performance of various operations. The operations include, during a first timing interval, storing, in a compute cell of the PiM block, a first weight value obtained from the storage bank and a first input obtained from the input register; and performing a first computation at the PiM block using the compute cell, the first weight value, and the first input. The operations further include, during a second timing interv al that overlaps with the first timing interval, storing, in the compute cell of the PiM block, a second, different input obtained from the input register; and performing a second computation at the PiM block using the compute cell, the first weight value, and the second input.

[0015] Other implementations of this and other aspects include corresponding sy stems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardw are, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0016] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0017] Relative to prior approaches, the disclosed techniques improve multi-batch processing by leveraging weight reuse and unique timing intervals to implement per batch pipeline stages efficiently and concurrently in a PiM block of an integrated memory device. For example, clock cycles for data access / fetch timing intervals and compute timing intervals of the pipeline stages can be staggered to efficiently support multi-batch computations such that operands for one computation can be fetched concurrent with execution of computations that involve other operands. The disclosed techniques can, therefore, be used to further optimize data access speeds and maximize memory bandwidth within an integrated memory device, which are useful advantages for enhancing overall workload performance of the disclosed PiM architecture.

[0018] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Fig. 1 is a block diagram of an example computing system w ith at least one SoC.

[0020] Fig. 2 shows an example PiM architecture with corresponding compute elements.

[0021] Fig. 3 is an example integrated system for executing multi-batch computations in a memory device.

[0022] Fig. 4 shows example approaches for executing multi-batch computations using the integrated system of Fig. 3.

[0023] Fig. 5 shows an example architecture of a compute cell inside a computational array or matrix multiplication unit of a PiM block.

[0024] Fig. 6 is an example process for executing multi-batch computations in a PiM block of a memory device.

[0025] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0026] Fig. 1 is a block diagram of an example computing system 100 that includes a system-on-chip 102 (“SoC 102”). The SoC 102 includes a central processing unit 104 (“CPU104”), a memory controller 105, a shared memory 106 (“memory 106”), a resource manager 108, and an IP / circuit block 110. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at system 100.

[0027] The CPU 104 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0028] The memory 106 is a system memory7, shared memory , or both. In the example of Fig. 1, memory 106 is depicted external to circuit block 110. However, memory 106 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory7106 can be random access memory of the SoC 102, such as static random-access memory7(SRAM), dynamic random access memory7(DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0029] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit 110. The memory 106 can also include various other types of memory7, such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), or a combination of these.

[0030] The resource manager 108 is implemented in hardware and software. Aspects of the resource manager 108 can be also implemented as firmware of the SoC 102 or firmware of a device of the SoC 102, such as a DRAM memory device or the CPU 104. The resource manager 108 is a processor-in-memory (PiM) resource manager (“PiM resource manager 108”) that includes control logic implemented in hardware, software, or both. For example, the PiM resource manager 108 can include resources such as flip-flops, registers, buffers, etc. that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software.

[0031] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 1 10can include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, TPU 114, DSP 116, and GPU 118 can be a respective proprietary' IP block (or IP device) of a particular entity or device manufacturer.

[0032] One or more aspects of the PiM resource manager 108 can be implemented as a software routine (or module) of the CPU 104, which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 102. such as memory 106. In some implementations, each processor (e.g., ISP 1 12, DSP 116, TPU 114, GPU 118) of the SoC 102 includes multiple cores and the CPU 104 and / or the PiM resource manager 108 can generate control signaling 124 to manage and distribute memory intensive compute operations to a memory device 122 (e.g., DRAM) to minimize the processing load at each core of the processors. The control signaling 124 is routed at system 100 using an example bus 120 of the SoC 102. The control signaling 124 can include commands, requests, data, instructions, or combination of these.

[0033] The PiM resource manager 108 cooperates with the CPU 104, memory' controller 105 and storage controller 107 to dynamically control and manage one or more compute-inmemory (CIM) operations. In some implementations, the CIM operations are executed at the SoC 102 in support of a heterogeneous compute operation between two or more processing units that are included among the IP block 110, the CPU 104, or both. More specifically, the PiM resource manager 108 is configured to generate control signaling 124 and use one or more discrete signal values of the control signaling 124 to manage and boost data access operations at the memory device 122.

[0034] The system 100 includes an example memory device 122. The memory7device 122 can include multiple memory' dies. For example, the memory' device 122 can include N memory die, where A is an integer greater than 1. The memory device 122 can be a dynamic random-access memory (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory' device 122 is configured to perform or support various ty pes of PiM operations, CiM operations, and memory' -near-computing operations (“MnC operations'’). The memory device 122 performs or supports these operations using its multiple PiM compute elements, which are described below with reference to at least Figs. 2- 5.

[0035] The SoC 102 cooperates with the memory device 122 to perform computations across one or more bank groups of the memory device 122. The computations can be for operations or workloads that involve one or more of the processors at IP block 110. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In at least one example the memory device 122 may be external to the SoC 102, whereas in another example the memory device 122 may be internal to the SoC 102. In some implementations, the SoC 102 is integrated (or co-located) with the memory device 122, for example, as distinct circuit die(s) that are co-located in a single integrated circuit package.

[0036] In the example of Fig. 1, system 100 and the SoC 102 is an integrated circuit of an example user / client device 130, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 130a, tablet 130b, laptop 130c, smartwatch or wearable device 130d. The devices 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0037] Fig. 2 shows an example processor-in-memory (PiM) architecture 200 for boosting PiM data access performance based on control signals generated using the SoC 102, the memory device 122, or both. In the example of Fig. 2, the memory device 122 includes a first memory die-1 with a first bank group that has multiple memory banks, where each memory bank includes one or more memory arrays and a second memory die-2 with a second bank group that has multiple memory banks, where each memory' bank includes one or more memory' arrays. In some implementations, the PiM architecture 200 includes multiple bank groups, multiple memory’ die, or both. For example, a single memory’ die can include multiple bank groups and / or multiple bank groups can be distributed across multiple memory die.

[0038] The PiM architecture 200 includes multiple PiM blocks 202, where each PiM block 202 includes multiple compute elements. For example, a first PiM block 202 of PiM architecture 200 includes mode register 204-1 and process unit 206-1, whereas a second, different PiM block 202 of PiM architecture 200 includes mode register 204-2 and process unit 206-2. Each process unit 206-1, 206-2 can include a processor, a processor unit, or a processor core, such as a CPU. Each process unit 206-1, 206-2 can also include an example computation unit such as an arithmetic logic unit (ALU) or multiply-accumulate cell (MAC).

[0039] In some implementations, the PiM architecture 200 is included in the memory device 122 as multiple discrete integrated circuits, where each integrated circuit is local to a given memory die (e.g., die-1, 210-1, and die-2, 210-2) and interacts or communicates with arrays of memory cells at that memory die. For example, the PiM architecture 200 can include compute elements that are replicated and distributed across each of the memory die in the memory device 122. In some other implementations, the PiM architecture 200 is included in the memory device 122 as a single integrated circuit that interacts or communicates with each memory' die of the memory' device 122, including the arrays of memory cells at each memory' die.

[0040] To boost PiM data access performance as described in this specification, the PiM blocks 202 or process units in the PiM architecture 200 are located within the memory device 122 but outside of a section of the memory device 122 that includes the bank groups. The section may be defined as a discrete memory die or defined in some other way (e.g., a portion of a memory die). Irrespective of the hardware configuration or layout of PiM architecture 200, the PiM blocks 202 are sufficiently external to the bank groups such that the PiM blocks 202 communicate with the bank groups based on a particular timing constraint that can be leveraged to boost PiM data access performance with cross bank group data aggregation.

[0041] The PiM operations can include standard CPU functions, whereas the CiM operations and MnC operations can include standard arithmetic operations, such as computations normally performed by an ALU or MAC. The CiM operations and MnC operations can also include computational functions of a TPU 1 14, such as multiplication and addition operations for matrix math, vector computations, linear algebra, and dot-product accumulations. In some implementations, each of the PiM operations, CiM operations, and MnC operations are performed in support of machine-learning computations, neural network computations, or both.

[0042] In some implementations, the PiM operations are an extension of the computational functions of the TPU 114. For example, a PiM block 202 can generate accumulated values from sets of weight values / inputs and activation inputs obtained from memory banks of different bank groups based on the TCCD_S timing constraint. The accumulated values are generated based on neural network computations performed using a computational array of the PiM block 202. The computational array can be a matrix multiplication unit with compute cells that are arranged as a systolic array. The accumulated values can be dot products of the sets of weight values and the activation inputs. That is, fora set of weights, the PiM block 202 multiplies each weight with each activation input and sums the products together to form an accumulated value.

[0043] The PiM architecture 200 can include a register or other portion of memory for storing data for a respective memon die or group of memory banks. For example, the data can be mode / configuration values. The data can also describe errors that occurred during a compute operation at a corresponding PiM block 202 of the memory device 122, or both. In some implementations, the register or other portion of memory is used to store configuration information, or associated instructions, for configuring aspects of a PiM block, or respective memory die, group of memory' banks, or a combination of these.

[0044] For example, the mode registers 204-1, 204-2 can be used to control or trigger selection of a particular mode in a PiM architecture, such as an error-capture mode, interleave configuration mode, multi-batch processing mode, etc. In some implementations, a particular mode is selected based on bit values of the mode registers 204-1, 204-2. For example, to trigger or select an interleave configuration mode(s) or multi-batch processing mode(s), a single bit, or a sequence of bits, can be defined for use in the mode register.

[0045] Using the disclosed techniques, data access operations are optimized by reading memory cells of bank groups at a frequency that exceeds the read command frequency afforded by a long delay timing constraint, such as the TCCD_L timing constraint. For example, an internal controller of the PiM block (1), 202-1, or PiM block (2), 202-2, can execute successive read commands at a frequency that is based on a clock cycle generated by the memory device 122. The internal controller can operate based on a particular clock frequency, e.g., a 200 or 800 MHz clock or 1000 MHz clock. Other clock frequencies are also within the scope of this disclosure. An example of an internal controller is described below with reference to the example of Fig. 3.

[0046] A frequency of the memory device 122 can be measured in megahertz (MHz) and indicates a number of cycles the memory device 122 (e.g., DRAM) can perform in one second, representing the number of data transfers that can occur between, for example, a PiM block 202 and bank group (or memory bank) of the memory’ device 122 in a given time frame. Generally, higher DRAM frequencies result in faster data accesses (read operations) as well as enhanced or accelerated pipeline computations and overall system performance.

[0047] An internal controller (e.g., controller 302, described below) of the PiM architecture can leverage the TCCD_S timing constraint to execute cross bank group accesses at nanosecond (ns) intervals, rather than the slower intervals associated with the TCCD L timing constraint. As used in this document, TCCD_S is a minimum delay required betweensuccessive column commands issued to different bank groups within a DRAM device, such as memory device 122. In general, each of TCCD S and TCCD L is a latency / period. In some implementations, the latency or delay period may be measured in clock cycles.

[0048] The time delay imposed by a short delay timing constraint (TCCD S) can be Vi the time delay imposed by the long delay constraint (TCCD_L). In view of this, the disclosed techniques can leverage cross bank group data access using the TCCD_S delay between successive column commands issued to distinct bank groups to implement higher frequency data access operations (relative to TCCD_L) by reducing the access timing by !4 based on the relationship: (TCCD S = ! * TCCD L). By way of example, a delay imposed by the TCCD_S and TCCD_L timing constraints can be defined using clock cycles (tex). For instance, the TCCD S constraint can impose a four (4 tex) delay, whereas the TCCD L imposes a constraint that is greater than the TCCD S constraint. For example, the TCCD L constraint can be an eight (8 RK) cycle delay period. The data access operation can be represented by a read command signal triggered by instructions processed at the PiM architecture of the memory’ device 122.

[0049] Fig. 3 is an example integrated system 300 ("system 300”) for executing multibatch computations in a memory' device. The system 300 includes a controller 302 that has corresponding control logic 304 for generating control signals that cause components of the system 300 to perform one or more functions associated with multi-batch computations (described below). In some implementations, system 300 is a sub-system of system 100 and represents an improved PiM architecture that enhances performance and energy efficiency of matrix x matrix operations for PiM operations in a DRAM device, such as memory' device 122.

[0050] The system 300 also includes an input register 306, a storage bank 308, and a computational unit 310. The input register 306 is for storing input samples (e.g., pixels, words, or activation values) and the storage bank 308 is for storing sets of weights 309 for a neural network layer through which an input sample is processed to generate a corresponding layer output. In some implementations, the computational unit 310 is included in a PiM block 202 of the memory' device 122. In some other implementations, the computational unit 310 is accessible by the PiM block 202 but is located outside of an area identified as a PiM block 202.

[0051] The computational unit 310 includes multiple compute cells 312. In some implementations, the computational unit 310 includes A number of cells 312. where A is an integer greater than 1. This definition for A also applies to other examples and elements ofsystem 300 (and system 100). Cell 312 includes one or more multiply accumulators (MACs) that perform arithmetic operations such as multiplication and addition. For example, each MAC of a cell 312 can be used to compute dot products and related matrix multiplications that are performed to convolve an input tensor of image pixel values with a weight tensor (e.g., a kernel fdter of weights). In some implementations, the computations are convolution operations performed to generate an output for a convolutional neural network layer. An example architecture of a compute cell 312 is described in more detail below with reference to Fig. 5.

[0052] As indicated above, cell 312 can be one of multiple cells in a computational unit 310 of a PiM block 202. In some implementations, the computational unit 310 includes an array of cells 312 and the PiM block 202 can be configured to supplement or complement the computational capacity of example special-purpose processor, such as a tensor processing unit (TPU) or neural network processor that executes an ML model to process an inference workload. The system 300 leverages timing interval controls, software hints for multi-batch processing, and computational unit 310 to enlarge the computational scope of the PiM architecture from a single batch operation to multi-batch operations.

[0053] For example, the system 300 implements the disclosed multi-batch processing techniques to decouple the data read and compute timing intervals to introduce a shorter timing / interval for executing machine-learning computations in DRAM devices such as memory device 122. The techniques leverage this timing modification to pipeline computations in support of concurrent / parallel batch operations. The software instructions (and some hardware aspects) of the control logic 304 for controller 302 are adapted to include a multi-batch support mode used to control the parallel batch operations.

[0054] Leveraging these techniques, when a large inference workload (e.g., LLM or GenAI task) is assigned to the TPU 114, the SoC 102 can offload a portion of the inference computations to the computational unit 310 of the PiM block 202, thereby expanding or extending the computational capacity of the TPU 114. In some implementations, this inference processing can involve computing convolutions, e.g., group convolutions, where data values and / or operands for an input sample, such as words or pixels, and weight values of a neural network layer are routed to cells 312 of computational unit 310 to execute the convolution based on computations performed at the cells. Each weight value can be represented using 2 bytes (2B). More specifically, each weight value can be a 16-bit Bfloat number (BF16) or a 32-bit floating point number (FP32).

[0055] The memory device 122 can include multiple PiM blocks 202 and each PiM block can be configured to receive control signals from the controller 302. The PiM blocks 202 are also configured for data communication with the input register 306 and the storage bank 308. The controller 302 can generate control signals 316 for controlling or triggering computations at cell 312 and for passing or broadcasting operands from the input register 306 and the storage bank 308. Each PiM block 202 can optionally include an input activation bus that couples the input register 306 to a computational unit 310 that includes multiple cells 312 and MACs. The controller 302 can generate control signals 316 to obtain weight values stored at the storage bank 308 of an example memory bank or bank group, which is described in more detail below with reference to Fig. 4.

[0056] The cells 312 and corresponding MACs of the computational unit 310 perform computations that produce output values that are routed to an output register 314-1, 314-2. In some implementations, the control logic 304 is configured to alternate selection of input registers 306 and output registers 314-1, 314-2 for corresponding batches of inputs. For example, the output values can be accumulated values that correspond to a particular batch operation and each output register 314-1, 314-2 can be configured to receive output (or accumulated) values for a particular batch (e g., batch 1 and batch 2). Each cell 312 uses its MAC to compute or calculate a product from an input and a corresponding weight value.The product may be a constituent part of a larger computation across a given input dimension to produce a set of output values for a corresponding output dimension.

[0057] In some implementations, the MACs of computational unit 310 are used to generate partial sums that are accumulated to generate a set of accumulated values for the corresponding output dimension. The MACs can be configured to compute a respective multiplication of different sets of operands and to add (or sum) the respective products of the different operand sets. For example, a MAC can compute a first product by multiplying a first activation value from input register 306 with a first weight value from storage bank 308. The MAC can also compute a second product by multiplying a second activation value from input register 306 with a second weight value from storage bank 308.

[0058] The MAC can then accumulate the two products by adding the first product and the second product. The addition operation performed on the first and second product can generate a partial sum. Some machine-learning computations involve the accumulation of multiple partial sums (e.g.. tens or hundreds of partial sums) to generate a set of accumulated values. In some implementations, multiplication of a set of operands produces partial sums for a particular output feature or activation, where the partial sums can be added to generatean output corresponding to a particular output feature. In some implementations, the memory device 122 includes an activation / linear unit that aggregates a set of accumulated values into the final linear output and applies an activation function (e.g., ReLU) to get a set of activated values.

[0059] In some implementations, an optional single or double data buffer 320 is included with the PiM architecture to hold, stage, or otherwise buffer weight values obtained from the storage bank 308. For example, a single data buffer 320 can be used to buffer weight values 322, whereas a double data buffer 320 can be used to buffer weight values 322 and 324. The buffer 320 can be leveraged to support overlapping DRAM read and compute pipelining operations.

[0060] The system 300 is configured to implement a two-stage compute pipeline, where a first stage 330 is used to process a first batch of inputs obtained from the input register 306 and a second stage 332 is used to process a second batch of inputs. A first compute interval of the PiM block 202 is used to process ML inputs of the first batch of inputs and a second, different compute interval of the PiM block 202 is used to process ML inputs of the second batch of inputs. Thus, each of the first and second compute intervals correspond to the different pipeline stages 330, 332 of the PiM block 202 and include a subset of overlapping clock cycles. The first stage 330 and the second stage 332 are described in more detail below with reference to Fig. 4.

[0061] Fig. 4 shows example approaches or techniques 402, 404 for executing multibatch computations using the integrated system 300 of Fig. 3.

[0062] Technique 402 leverages an additional MAC group (“MAC Group 2”) by expanding the computational unit 310 to include additional compute cells 312 and MAC groups / resources. When technique 402 is used, the data read and compute intervals remain unchanged, such that a computation at cell 312 and a data read from memory device 122 (e.g., DRAM) uses the same DRAM timing latency. Thus, the memory device 122 timing for reading data from memory banks of the memory device 122 is also used as the determined timing of the parallel MAC operations in the PiM block 202.

[0063] In other words, the data read and compute timing intervals are not decoupled and instead operate based on the same clock signals and timing interval. When the data read and compute operations use the same timing, additional MAC / compute resources are required to implement technique 402 for multi-batch computations. For example, a duplication of resources, e.g.. an additional MAC group, is required at a PiM block to enable parallel batch processing. While useful, the technique 402 increases resource and area overhead at the PiMblock and does not allow for leveraging certain area and computational efficiencies associated with data reuse of weight values.

[0064] System 300 leverages technique 404 to execute multi-batch computations in memory device 122. As described above, system 300 is configured to implement a two-stage compute pipeline, where inputs of at least two different batches can be processed in parallel at the PiM block. During a first compute interval 406 of a first stage 330, the PiM block processes ML inputs of a first batch of inputs. During a second compute interval 408 of a second stage 332, the PiM block processes ML inputs of the second batch of inputs. Thus, each of the first and second compute intervals correspond to the different pipeline stages 330, 332. For example, the technique 404 for executing multi-batch computations in memory device 122 includes implementing unique and overlapping data read intervals and computation intervals using resources of the PiM blocks.

[0065] The computation intervals include at least two distinct compute pipeline intervals 406, 408, which are staggered to enable concurrent processing of respective batches of inputs. More specifically, the first and second compute intervals 406, 408 include a subset of overlapping clock cycles. In some implementations, timing aspects of the pipeline stages 330, 332 are staggered to efficiently support multi-batch computations. For example, the compute intervals 406, 408 for the different pipeline stages can be staggered using a one clock cycle offset to support data (e.g., weight value) reuse in a manner that enables efficient, concurrent processing of inputs across multiple input batches.

[0066] In some implementations, each of techniques 402, 404 also includes generating control signaling at the SoC to configure aspects of the multi-batch computation and to trigger (or initiate) a multi-batch compute mode of the PiM block. The control signal can also be generated locally at the memory device 122 using control logic 304 of the controller 302 that communicates with the PiM block. The multi-batch compute mode is triggered to process inputs of two or more different batches of inputs (e.g., batch 1 , batch 2, .. . batch N) at least by reusing a weight value as an operand in multiplication operations that are performed to compute ML inference outputs.

[0067] For example, the SoC 102 (or memory device 122) can generate control signals and initiate a multi-batch compute mode at the PiM block based on the control signals. During the multi-batch compute mode, the memory device 122 uses the PiM block(s) to concurrently process inputs of at least two distinct batches of inputs obtained from the input register 306. For example, in response to initiating the multi-batch compute mode, the PiMblock concurrently processes inputs from different batches of inputs based on computations that are performed by reusing one or more weight values.

[0068] In the example of Fig. 4, during the multi-batch computation mode, a first multiplication operation is performed at a compute cell 312 to multiply a first input of batch 1 and a first weight value during the first compute pipeline interval 406. This first multiplication operation is performed for the first pipeline stage 330 at an initial clock cycle 412 using a MAC 410 of the compute cell 312. As indicated at Fig. 4, arithmetic circuitry of MAC 410 includes an example multiplication circuit (x) and an example addition circuit (+). The MAC 410 generates a first product based on the first multiplication operation. The MAC 410 can then perform a first addition operation at a subsequent clock cycle 414. The first addition operation is performed to accumulate the first product with a different product, such as a product that was generated during a prior compute interval and / or during a prior compute cycle.

[0069] In some implementations, circuitry of the computational unit 310 is scaled (or scalable) based on the number of pipeline stages. A quantity of MAC units in the computational unit 310 can be scaled to include a required number of MACs to support computations for each pipeline stage. For example, system 300 can include N pipeline stages for processing N input batches, and the multiplication circuitry7of computational unit 310 can be scaled or adapted to provide the requisite compute capacity for each pipeline stage. For A pipeline stages, the computational unit 310 can be scaled to include A multipliers per MAC or A MACs per pipeline, if each MAC is configured to include a single multiplier. Differing combinations of multipliers and / or MACs can be used to provide the required computational capacity7for each of the N pipeline stages.

[0070] Additionally, during the multi-batch computation mode and concurrent with at least the first addition operation of the first pipeline stage 330, a second multiplication operation is performed at the compute cell 312 to multiply a second input and the first weight value during the second compute pipeline interval 408. This second multiplication operation is performed for the second pipeline stage 332 at an initial clock cycle 414 (of that the second stage) using the MAC 410 of compute cell 312. As indicated at Fig. 4. the first multiplication operation for batch 1 that was initiated at clock cycle 412 (for stage 1) is complete at or before the triggering of clock cycle 414 at the second stage. This means that, at clock cycle 414, the multiplication circuit (x) of MAC 410 is available to perform the second multiplication operation for an input of batch 2 at the second pipeline stage 332.

[0071] The MAC 410 generates a second product based on the second multiplication operation. The MAC 410 can then perform a second addition operation at a subsequent clock cycle 416. Notably, the first addition operation for batch 1 that was initiated at clock cycle 414 (for stage 1) is complete at or before the triggering of clock cycle 416 at the second stage. This means that, at clock cycle 416, the addition circuit (+) of MAC 410 is available to perform the second addition operation for an input of batch 2 at the second pipeline stage 332. The second addition operation is performed to accumulate the second product with a different product, such as product that was generated during a prior compute interval and / or during a prior compute cycle of stage 2. In some implementations, the second product of stage 2 can be accumulated with the first product of stage 1.

[0072] In the example of Fig. 4, the first input and the first weight value were routed to, and stored in, the compute cell 312 during a prior data read interval. In some implementations, the respective data read interval for each pipeline stage 330, 332 coincides with two compute cycles 407, 409 of the corresponding compute intervals 406, 408 of that stage. In some other implementations, the durations indicated as compute cycles 407, 409 are described alternatively as compute intervals 407, 409. and intervals 406, 408 are described as data read intervals 406, 408. Further, as shown at Fig. 4, the compute interval / cycle 407 can correspond to a first pipeline stage 330, whereas the compute interval / cycle 409 can correspond to a second, different pipeline stage 332.

[0073] Relatedly, each of pipeline intervals 406, 408 can include N compute cycles, such that the compute cycles of the pipeline intervals 406, 408 are a ratio of the data read interval. In some implementations, if the data read interval = a, then the compute pipeline interval (or a compute cycle of the interval) = a / N, which can also indicate the number of pipeline stages. In the example of Fig. 4. each of pipeline intervals 406, 408 includes at least two compute cycles, such that each compute cycle is ’A of the data read interval, a. Each compute cell 312 includes a threshold amount of storage that permits a corresponding threshold amount of weight reuse.

[0074] The system 300 can execute multi-batch processing with improved efficiency over prior approaches at least by reusing weight values for a given neural network layer on multiple inputs of different batches. In particular, the w eight values are reused a number of times such that a compute time of reusing the w eight inputs multiple times is less than a fetch time of accessing / obtaining new w eight values from a memory bank of the memory device 122, which allows for maximizing throughput in the circuits of system 300. The control logic 304 can determine a weight reuse value that specifies an amount of reuse of a given weightvalue. In some implementations, both the weight values and values for input samples can be reused by other cells in the computational unit 310 at a subsequent clock cycle.

[0075] Technique 404 leverages the local storage capacity of the compute cells 312 to maximize reusing one or more weight values across two or more batches, such that processing of a first input of a first batch overlaps with processing of a second input from a second, different batch. Thus, the same weight value(s) can be reused with different activation inputs across two or more batches of inputs. In particular, the PiM block reuses the weight values by computing, at the computational unit 310, accumulated values using the same weight value stored at MAC 410 and different activation inputs of batch 1 and batch 2. Technique 404 leverages data reuse to provide a unique data processing technique for concurrently processing two or more batches without expanding the circuit area of the computational unit 310. Thus, technique 404 represents an area efficient batch processing technique.

[0076] The system 300 has a pipeline processing / compute latency that is shorter than a corresponding data read latency, such that computations at a compute cell 312 can be performed in less time (or fewer clock cycles) than is required to obtain data from registers, memory banks, or bank groups of the memon device 122. Data read intervals of the PiM block can execute read / access operations based on the TCCD_L timing constraint. In some implementations, using control signals generated by the control logic 304, the system 300 (or PiM block) can be configured to: i) execute read / access operations based on the TCCD L timing constraint during a normal mode of operation and ii) execute pipelining processing / compute operations based on the TCCD_S timing constraint during the multibatch processing mode of operation. A PiM configuration with multiple bank groups can use the TCCD S minimum delay for executing successive read commands against bank groups. In this case, a shorter timing will be required to enable the pipeline processing.

[0077] Relatedly, each read interval of the PiM block can operate based on a first frequency, I, whereas each compute pipeline interval of the PiM block can operate based on a second frequency, fl, that is different from the first frequency, / ! . In some implementations, / ! and fl are clock frequencies and fl is at least A of / I. Hence, system 300 has a processing / compute speed and a speed at which it can access data in its memory cells, where each speed is based on a corresponding clock frequency. In these implementations the processing / compute speed is faster than the memory access speed. For example, the compute speed (based on fl) can be at least 2x faster than the memory access speed (based on / !). By reusing weight values, system 300 can leverage the compute speed along with the thresholdstorage capacity of each compute cell 312 to perform meaningful computations while, in parallel, executing memory accesses to obtain weight values (or inputs) for use in a subsequent computation.

[0078] Fig. 5 shows an example architecture 500 of a compute cell inside a computational array or matrix multiplication unit of a PiM block. For context, each PiM block can include an array of compute cells, where each compute cell is configured to perform different operations concurrently. An individual compute cell may be referred to alternatively as a MAC or more generally as a “cell”. As shown in the example of Fig. 5, each compute cell can include certain hardware features and data paths for multiplication and addition operations, with some intermediate data storage (e.g. registers) to facilitate these operations.

[0079] In architecture 500 the example compute cell can include an activation register 506 that stores an activation input. The activation register can receive the activation input from input register 306, from a memory bank of memory device 122, or from another compute cell, such as an adjacent cell in an array of compute cells. The compute cell can include a weight register 502 that stores a weight input. The weight input can be transferred from a memory bank of memory device 122, from a weight fetch er interface of the PiM architecture, or from another compute cell, such as an adjacent cell in an array of compute cells. The cell can also include a sum in register 504. The sum in register 504 can store an accumulated value received from another compute cell, e.g., an adjacent cell. Multiplication circuitry 508 can be used to multiply the weight input from the weight register 502 with the activation input from the activation register 506. The multiplication circuitry 508 can output the product to summation circuitry' 510.

[0080] The summation circuitry can sum the product and the accumulated value from the sum in register 504 to generate a new accumulated value. The summation circuitry 510 can then send the new accumulated value to another sum in register located in another compute cell, e.g., an adjacent cell. The new accumulated value can be used as an operand for a summation operation in the receiving compute cell. The compute cell can also shift the weight input (512) and the activation input (514) to adjacent cells for processing. For example, the weight register 502 can send the weight input to another weight register in a bottom adjacent cell in the array of cells. The activation register 506 can send the activation input to another activation register in the right adjacent cell. Both the weight input and the activation input can therefore be reused by other cells in the array at a subsequent clock cycle.

[0081] In some implementations, the cell also includes a control register. The control register can store a control value corresponding to a control signal that is used to determine whether the compute cell should shift either the weight input or the activation input to another compute cell. In some implementations, shifting the weight input or the activation input takes one or more clock cycles. The control signal can also be used to determine whether the activation input or weight inputs are transferred to the multiplication circuitry 508, or whether the multiplication circuitry 508 operates on the activation and weight inputs. The control signal can also be passed to one or more compute cells, e.g., adjacent cells, via a signal trace.

[0082] In some implementations, weights are pre-shifted into a weight path register 512. The weight path register 512 can receive the weight input, e.g., from another compute cell, and transfer the weight input to the weight register 502 based on the control signal.The weight register 502 can statically store the weight inputs for reuse at the compute cell such that as activation inputs are transferred to the cell, e.g., through the activation register 506, over multiple clock cycles, the weight input remains within the cell and is reused for computations as the cell. Therefore, the weight input can be applied or reused with multiple activation inputs, e.g., using the multiplication circuitry 508, to compute accumulated values that are passed to summation circuit 510 or another compute cell.

[0083] Fig. 6 is an example process 600 for executing multi-batch computations in a PiM block of a memory device, such as PiM block 202 and memory device 122. Process 600 is implemented or executed at system 100 using at least the SoC 102 and memory device 122 described above. Hence, descriptions of process 600 will reference the above-mentioned computing resources of system 100. In some examples, the steps or actions of process 600 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0084] Referring again to process 600, during a first timing interval, the memory device 122 stores a first weight value and a first input in a compute cell of a PiM block 202 in the memory device (602). For example, the first weight value is obtained from a storage bank 308 of the memon device 122, whereas the first input is obtained from an input register of the memory device 122. The first input corresponds to a first batch of inputs. The first weight value and the first input are stored during a data access / read interval of the first pipeline stage 330. The memory device 122 performs a first computation at the PiM block 202 using a MAC of the first compute cell, the first weight value, and the first input (604).

[0085] During a second timing interval that overlaps with the first timing interv al, the memory device 122 stores a second, different input in the compute cell of the PiM block 202 of the memory device (606). The second input corresponds to a second, different batch of inputs. In some implementations, each of the first and second inputs are neural network inputs that are processed to generate machine-learning outputs for a GenAI workload. The second input is also obtained from the input register of the memory device 122 and is stored during a data access / read interval of the second pipeline stage 332.

[0086] The memory device 122 performs a second computation at the PiM block 202 using the compute cell, the first weight value, and the second input (608). The second computation is performed during a portion of the second timing interval that overlaps with the first timing interval. Notably, the second computation is performed using the same compute cell (and / or MAC) used to perform the first computation, but is associated with a different pipeline stage of the PiM block. Leveraging a single MAC or compute cell 312 to concurrently process inputs of different batches by performing computations within distinct pipeline stages corresponds to the advantages in area efficiency offered by technique 404. For example, the weight reuse, overlapping compute intervals, and associated clock signals are used to efficiently perform the computations without the increase in area / circuitry to include additional MACs within the PiM architecture.

[0087] The first computation can include a first addition operation associated with batch 1 of the first pipeline stage 330, whereas the second computation can include a second multiplication operation associated with batch 2 of the second pipeline stage 332. In this example, the first addition operation and the second multiplication operation can be performed concurrently based on a clock cycle of a first timing interval 406 that overlaps with a clock cycle of a second timing interval 408. That is, the first addition operation and the second multiplication operation are performed concurrently based on clock cycle 414, which is an overlapping clock signal of intervals 406, 408 as well as intervals 407, 409. In at least one example embodiment the first computation and the second computation are performed by different MACs within a compute cell 312 or by different compute cells 312 within computational unit 310.

[0088] In some implementations, a particular PiM block 202 may be tasked with executing a portion of a convolution computation for a neural netw ork layer. For example, based on its tasking, the PiM block can perform multiplication operations that use: i) an input or activation (e.g., a first operand) stored in a memory location of the memory device 122 and ii) a weight value (e.g., second operand) stored in a different memory location of the memorydevice 122. The multiplications are performed at each MAC of a given cell 312, using respective multiplication circuitry of each MAC.

[0089] In the examples described above, the inputs are stored in register 306, whereas the weights are stored in storage bank 308. However, other types of memory constructs may be used. For example, the operands may be stored across memory banks or bank groups of the memory device 122. In some implementations, the operands are stored across memory banks or bank groups in a contiguous or non-contiguous data layout.

[0090] The steps of process 600 can be also performed at a hardware integrated circuit as part of a larger compute operation to generate an ML output, including an output for a neural network layer of a neural network that implements one or more ML models. For example, the output can be a portion of a computation for a ML task or inference workload to generate an image processing, speech processing, or image recognition output. In some implementations, process 600 is executed to obtain data used to perform computations for generative artificial intelligence (“GenAI”) applications, such as an application that generates text responsive to a user query.

[0091] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory' program carrier for execution by, or to control the operation of, data processing apparatus.

[0092] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0093] The term '‘computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code thatcreates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0094] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0095] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0096] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as. special purpose logic circuitry, e g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0097] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0098] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memon devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0099] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid cry stal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0100] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0101] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0102] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0103] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0104] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:

1. A multi-batch processing method implemented using an integrated circuit comprising an input register, a processing-in-memory (“PiM”) block, and a storage bank, the method comprising: during a first timing interval: storing, in a compute cell of the PiM block, a first weight value obtained from the storage bank and a first input obtained from the input register; and performing a first computation at the PiM block using the compute cell, the first weight value, and the first input; during a second timing interval that overlaps with the first timing interval: storing, in the compute cell of the PiM block, a second, different input obtained from the input register; and performing a second computation at the PiM block using the compute cell, the first weight value, and the second input.

2. The method of claim 1 , further comprising: generating control signals using a control logic of the integrated circuit; initiating a multi-batch compute mode of the PiM block based on the control signals; and performing the first and second computations during the multi-batch compute mode of the PiM block.

3. The method of claim 2, further comprising: in response to initiating the multi-batch compute mode of the PiM block: concurrently processing inputs of at least two different batches of inputs by reusing the first weight value, such that processing of the first input overlaps with processing of the second input.

4. The method of claim 2 or 3, wherein performing the second computation comprises: performing the second computation during a portion of the second timing interval that overlaps with the first timing interval.

5. The method of any one of claims 2 to 4, wherein performing the second computation comprises: performing the second computation concurrently with performing the first computation.

6. The method of any one of claims 2 to 5, wherein: the first input obtained from the input register is from a first batch of inputs; and the second, different input obtained from the input register is from a second, different batch of inputs.

7. The method of claim 6, wherein: the first timing interval is a first compute pipeline interval of the PiM block, for the first batch of inputs; and the second timing interval is a second compute pipeline interval of the PiM block, for the second batch of inputs.

8. The method of claim 7, wherein performing the first computation comprises: performing, at the compute cell and during the first compute pipeline interval, a first multiplication operation to multiply the first input and the first weight value; generating a first product based on the first multiplication operation; and performing a first addition operation to accumulate the first product and a different product.

9. The method of claim 8, wherein performing the second computation comprises: performing, at the compute cell and during the second compute pipeline interval, a second multiplication operation to multiply the second input and the first weight value; generating a second product based on the second multiplication operation; and performing a second addition operation to accumulate the second product and a different product.

10. The method of claim 9, wherein: the first addition operation and the second multiplication operation are performed concurrently based on a clock cycle of the first timing interval that overlaps with a clock cycle of the second timing interval.

11. The method of any one of claims 7 to 10, further comprising: during a first read interval of the PiM block: i) obtaining the first weight value from the storage bank; and ii) obtaining the first input from the input register; and during a second read interval of the PiM block: i) obtaining the second input from the input register.

12. The method of claim 11, wherein each read interval of the PiM block executes read operations based on a TCCD_S timing constraint.

13. The method of claim 1 1 or 12, wherein: each read interval of the PiM block operates based on a first frequency, / !; and each compute pipeline interval of the PiM block operates based on a second frequency, / 2, that is different from the first frequency, / I .

14. The method of claim 13, wherein / 2 is at least Vi of / ! .

15. The method of any one of claims 1 to 14, further comprising: performing computations for a plurality of inputs using a plurality of pipeline stages of the PiM block; and wherein each input of the plurality of inputs corresponds to a distinct batch and each batch corresponds to a distinct pipeline stage of the plurality of pipeline stages.

16. A system comprising: a processing device; an integrated circuit comprising an input register, a processing-in-memory (“PiM”) block, and a storage bank; and a machine-readable storage device for storing instructions that are executable by the processing device to cause performance of operations comprising: during a first timing interval: storing, in a compute cell of the PiM block, a first weight value obtained from the storage bank and a first input obtained from the input register; andperforming a first computation at the PiM block using the compute cell, the first weight value, and the first input; during a second timing interval that overlaps with the first timing interval: storing, in the compute cell of the PiM block, a second, different input obtained from the input register; and performing a second computation at the PiM block using the compute cell, the first weight value, and the second input.

Images