Interleaving commands and data writes to a processing-in-memory architecture to optimize execution of in-memory computations

Abstract

Methods and systems for hardware and software techniques for interleaving compute commands and input data write operations to a PiM module to optimize execution of in memory computations. A PiM compute module executes the interleaved commands to perform multiply and accumulate (MAC) operations using parameters read from a memory array and input data read from an input queue. A memory controller interleaves the commands and input data writes to the PIM module based on the duration of the MAC operations and / or a timing parameter for issuing write commands or for consuming data from the input queue. The memory controller can interleave different types of commands concurrent with interleaving the commands with the input data write operations. The interleaving operations are timed to maintain a threshold quantity of input data in the input queue to minimize (or prevent) data underflow or overflow during execution of the MAC operations.

Description

INTERLEAVING COMMANDS AND DATA WRITES TO A PROCESSING-IN-MEMORY ARCHITECTURE TO OPTIMIZE EXECUTION OF IN-MEMORY COMPUTATIONSBACKGROUND

[0001] This specification generally relates to memory devices used to execute computations.

[0002] Modern computing systems often incorporate a wide variety of compute processing units that each offer different computing capabilities and trade-offs. Efficient execution of a given compute job often involves parsing computations into meaningful subtasks or workloads that are mapped to available processor cores of a computing system. The computations may be parsed and mapped based on suitability criteria, such as processor capability’, performance, and power. Generally, this overall process of allocating portions of a computation to appropriate processor resources is referred to as heterogeneous computing.

[0003] At least one processor core of a System-on-Chip (“SoC”) can be an Intellectual Property’ block (“IP block”) that executes a respective portion of a computational operation for different multimedia workloads. An example use case can involve processing image or speech data captured respectively by a camera or microphone on the mobile device. The SoC can use a heterogeneous compute operation to process input samples derived from the image data, the speech data, or both. An example step in the heterogeneous compute operation can include processing data associated with the input samples using in-memory processing or computing capabilities.SUMMARY

[0004] This specification describes hardware and software techniques for storing data in a processor-in-memory' architecture (“PiM architecture”) used for memory' -bound compute workloads. The PiM architecture defines one or more PiM blocks of a memory device in data communication with an SoC that is separate from the memory device. Each PiM block includes compute elements, such as a processor unit, mode registers, and one or more arithmetic logic units (ALUs). For example, the PiM block can include discrete processors, processor units, register devices, buffers, etc. that cooperate to form one or more PiM compute elements.

[0005] Prior design approaches for storing data in a PiM architecture require opening a “dummy” page before writing data to a memory array in a dynamic random-access memory'(DRAM) or PiM register device. The dummy page must then be closed once the writing operation is complete. When writing data to a PiM register device, the steps of opening and closing a dummy page are unnecessary as the DRAM page is not being accessed. In contrast to these prior approaches, the techniques of this specification remove the unnecessary steps of opening and closing dummy pages when writing data to a PiM register device. Additionally, if the size of data being written is below a threshold value, further processing efficiency can be realized by transmitting the data to be written in a command bus of the PiM architecture.

[0006] This specification also describes hardware and software techniques for interleaving PiM compute commands and input queue write commands for input data write operations communicated to a PiM module (or register device) of the PiM architecture to optimize execution of in-memory computations. In some aspects, interleaving PiM commands with input queue write commands can refer to, in a sequence of commands, positioning one or more input queue write commands between at least tw o PiM commands or positioning one or more PiM commands between at least tw o input queue write commands. In some examples, input queue writes (also referred to as “input queue write operations’") can be interleaved with the PiM compute commands. For example, the input queue write operation can include loading input data onto a data bus (e.g., DQ bus 125 shown in Fig. 8), loading input data from the data bus to the input queue, or generally operations related to providing input data for processing by the PiM compute module, and this operation can be interleaved w ith the communication of the PiM compute commands to the PiM module. A PiM module executes the interleaved commands to perform multiply and accumulate (MAC) operations using parameters (e.g., weights for a neural network layer) read from a memory' array of the memory device and input data (e.g., image pixel data or word tokens) read from an input queue. A memory controller on the SoC interleaves the commands and communicates the interleaved commands to the PiM module based on the duration of the MAC operations and / or a predefined timing parameter that governs issuing input queue write commands and consuming input queue data.

[0007] The memory controller can interleave different types of commands concurrent with interleaving the PIM commands with the input data write operations, where the input data write operations are performed via write commands issued by a memory' controller of the SoC to memory device . For example, the memory controller can perform an interleave operation that interleaves an input queue write command with two or more PiM compute commands. Additionally, the memory controller can perform the next interleave operation that interleaves PiM commands with input data write operations that include providing inputdata to the PiM module. The first and second interleaved operations can be executed sequentially or concurrently in the PiM module.

[0008] The memory controller’s interleaving operations can be timed such that the commands and input data are transferred to the PiM module while the MAC operations are executed. Further, the interleaving operations are timed to maintain a threshold quantity of input data in the input queue, which minimizes (or prevents) data underflow or data overflow during execution of the MAC operations. More specifically, the timing of interleaving commands to the PiM module and input data write operations to the PiM module is configurable at the memory controller to optimize execution of in-memory computations for a given workload.

[0009] A timing parameter that controls interleaving commands and data writes to the PiM module can be predefined or dynamically configured to maintain sequential execution of MAC operations for larger input datasets. For example, the timing of interleaving PiM compute commands and input queue write commands can be configured such that a threshold quantity of input data is present in the input queue to maintain continued, sequential execution of MAC operations for completing a given inference workload. In some cases, the timing interval between providing an input queue write command and providing input data to the PiM module is dynamically determined for each workload or workload ty pe. For example, a timing parameter that defines this timing interval can be dynamically determined at the SoC such that interruptions or pausing of in-memory computations for a given workload do not exceed a threshold number of clock cycles (e g., 5 or 10 cycles).

[0010] In some aspects, a method performed by a controller of a system-on-chip (SoC) communicably coupled with a processing-in-memory (PiM) module, includes obtaining a control value indicating a duration of a compute operation to be executed at the PiM module; determining, based on the control value, at least one timing parameter used to communicate commands to the PiM module, the commands comprising a first command used to write input data to a memon queue coupled to the PiM module and a second command for executing compute operations; executing, by the controller and based on the timing parameter, an interleaving operation that interleaves commands for writing of input data to the memory queue with two or more of the commands for executing compute operations; and communicating, to the PiM module, the commands and input data based on the interleaving operation.

[0011] In some aspects, the method further includes communicating, to the PiM module, input data for pre-loading to the input / memory queue prior to execution of the computeoperation at the PiM module. In some aspects, no more than half the total amount of input data needed for executing all of the compute operations is communicated for pre-loading. In some aspects, the method further includes maintaining a counter indicating available input data in the input / memory queue; and communicating, to the PiM module, the input data based on the value of the counter. In some aspects, the at least one timing parameter indicates a duration between communicating the first command and the second command.

[0012] In some aspects, a system comprises one or more processors and one or more storage devices storing instructions that when executed by the one or more processors perform operations comprising: obtaining a control value indicating a duration of a compute operation to be executed at the PiM module; determining, based on the control value, at least one timing parameter used to communicate commands to the PiM module, the commands comprising a first command used to write input data to a memory queue coupled to the PiM module and a second command for executing compute operations; executing, by the controller and based on the timing parameter, an interleaving operation that interleaves the first command for writing of input data to the memory queue with two or more of the second command for executing compute operations; and communicating, to the PiM module, the commands and input data based on the interleaving operation.

[0013] In some aspects, the operations further include communicating, to the PiM module, input data for pre-loading to the memory queue prior to execution of the compute operation at the PiM module. In some aspects, no more than half the total amount of input data needed for executing all of the compute operations is communicated for pre-loading. In some aspects, the operations further include maintaining a counter indicating available input data in the memory queue; and communicating, to the PiM module, the input data based on the value of the counter. In some aspects, the at least one timing parameter indicates a duration between communicating the first command and the second command.

[0014] In some aspects, a method performed by a processing-in-memory (PiM) module coupled with a controller of a system-on-chip (SoC), includes receiving, from the controller via a command bus, an input queue write command interleaved with PiM commands; executing, by the PiM module, the PiM commands to retrieve parameters accessed from a memory array and input data received from the SoC; executing, at the PiM module, a multiply-accumulate (MAC) operation using the parameters and input data; and concurrent with executing the MAC operation, executing the input queue write command interleaved with the PiM commands to trigger receipt of additional input data from the SoC.

[0015] In some aspects, the method further includes receiving, from the controller via the command bus prior to receiving the PiM commands, a set of input queue write commands; responsive to receiving the set of input queue write commands, preloading input data received from the SoC; and executing at least a first PiM command of the received PiM commands based on preloaded input data. In some aspects, preloading input data includes preloading no more than half an amount of data needed for execution of all of the PiM commands. In some aspects, a write latency between a time when a write command is executed and a time when the associated input data is greater than a duration of execution of the PiM command.

[0016] In some aspects, a system includes one or more processors and one or more storage devices storing instructions that when executed by the one or more processors perform operations including: receiving, from the controller via a command bus, an input queue write command interleaved with PiM commands; executing, by the PiM module, the PiM commands to retrieve parameters accessed from a memory array and input data received from the SoC; executing, at the PiM module, a multiply-accumulate (MAC) operation using the parameters and input data; and concurrent with executing the MAC operation, executing the input queue write command interleaved with the PiM commands to trigger receipt of additional input data from the SoC

[0017] In some aspects, the operations further include receiving, from the controller via the command bus prior to receiving the PiM commands, a set of input queue write commands; responsive to receiving the set of input queue write commands, preloading input data received from the SoC; and executing at least a first PiM command of the received PiM commands based on preloaded input data. In some aspects, preloading input data includes preloading no more than half an amount of data needed for execution of all of the PiM commands. In some aspects, a write latency between a time when an input queue write command is executed and a time when the associated input data is loaded into an input data queue is greater than a duration of execution of the PiM command.

[0018] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0019] Removing the unnecessary steps of opening and closing dummy pages when writing data to a PiM register device drastically reduces wri te latency. These techniques also reduce switching latency when conducting writing operations across PiM memory die ranks that share the same command and data buses. In the scenario where the data to be written is small enough to be sent with the command bus, these techniques can broadcast the samewrite command to two ranks simultaneously - thus eliminating switching latency. A reduction in writing latency results in greater overall processing efficiency for the host device.

[0020] Interleaving input queue write commands for writing data to the input queue with PiM commands that trigger MAC operations can reduce the total time needed to transfer input data to a memory device, to complete computations on the PiM module, or both. In particular, prior approaches included a first duration for loading input data to the PiM module and a second duration for executing MAC operations using the input data loaded at the first duration. The first duration precedes the second duration such that any input data required for the MAC operation is loaded to the input queue in its entirely during the first duration (e g., an input data loading duration).

[0021] In contrast, by using the disclosed techniques for interleaving write commands that trigger operations for input data writes to the input queue with PiM commands that trigger MAC operations, a portion of the input data that would otherwise be loaded to the input queue during the input data loading duration is now loaded concurrently with executing the MAC operations at the PiM module. In this manner, at least a subset of the input data transfer operations of the first duration can now occur in parallel with the MAC operations of the second duration, rather than entirely sequentially with the second duration as in some prior approaches.

[0022] As a result, the input data loading duration is reduced for a given workload or task. Further, input data loaded during the MAC operation execution duration incurs no additional time. As indicated above, this is because the input data load operations are now carried out in parallel with the MAC operations. Thus, the total time for carrying out a set of operations for a given workload or task, which is the sum of the time required to load input data to the PiM module and the time required to execute MAC operations, is reduced, thereby improving the performance of the PiM architecture. In addition, as only a portion of all the input data is stored in the input queue, the overall size of the input queue can be reduced, thereby saving area, power, and cost associated with the input queue.

[0023] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 is a block diagram of an example computing system with at least one SoC.

[0025] FIG. 2 is an example prior method of writing data in a PiM architecture.

[0026] FIG. 3 is an example of an improved method of writing data in a PiM architecture.

[0027] FIG. 4 is an example of an improved method of writing data to two ranks in a PiM architecture.

[0028] FIG. 5 is an example improved method where data is sent via a command bus.

[0029] FIG. 6 is an example process for implementing the improved method of writing data in a PiM architecture.

[0030] FIG. 7 is an example process for sending data for writing via a command bus.

[0031] FIG. 8 is a block diagram of an example PiM module.

[0032] FIG. 9 is a flow diagram of an example process executed by a memory controller.

[0033] FIG. 10 is a timing diagram of an example sequence of commands provided by the memory controller to the PiM module.

[0034] FIG. 11 is a block diagram of an example memory controller.

[0035] FIG. 12 is a flow diagram of a process executed by the PiM module.

[0036] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0037] FIG. 1 is a block diagram of an example computing sy stem 100 that includes a system-on-chip, or host device, 102 (“SoC 102"). The SoC 102 includes a central processing unit 104 (“CPU 104"). a memory controller 105, a shared memory 106 (“memory 106”). a resource manager 108, and an IP / circuit block 1 10. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at system 100. The example computing system can implement the methods and processes described below with respect to FIGS. 2-7.

[0038] The CPU 104 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The applicationvalues may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0039] The memory 106 is a system memory, shared memory. or both. In the example of Fig. 1, memoiy 106 is depicted external to circuit block 110. However, memory 106 can include portions of memory7that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory of the SoC 102, such as static random-access memory (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0040] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory7that supports parallel access of its memoiy resources by two or more processors of the circuit 110. The memory 106 can also include various other types of memory, such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0041] The resource manager 108 is implemented in hardware and software. Aspects of the resource manager 108 can be also implemented as firmware of the SoC 102 or firmware of a device of the SoC 102, such as a DRAM memory device or the CPU 104. The resource manager 108 is a processor-in-memory (PiM) resource manager (“PiM resource manager 108”) that includes control logic implemented in hardware, software, or both. For example, the PiM resource manager 108 can include resources such as flip-flops, registers, buffers, etc. that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software.

[0042] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 1 16, and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary7hardware elements. For example, each of the ISP 112, TPU 114, DSP 116, and GPU 118 can be a respective proprietary7IP block (or IP device) of a particular entity or device manufacturer.

[0043] One or more aspects of the PiM resource manager 108 can be implemented as a software routine (or module) of the CPU 104, which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 102, such as memory 106. In some implementations, each processor (e.g., ISP 112, DSP116, TPU 114, GPU 118) of the SoC 102 includes multiple cores and the CPU 104 and / or the PiM resource manager 108 can generate control signaling via a command bus (“CA BUS”) 124 and data bus C'DQ BUS”) 125 to manage and distribute memory intensive compute operations to a memory device 122 (e.g., DRAM) to minimize the processing load at each core of the processors. The control signaling of the command 124 and data 125 buses is routed at system 100 using an example bus 120 of the SoC 102. The control signaling of the command 124 and data 125 buses can include commands, requests, data, instructions, or combination of these. In some examples, described below with respect to FIG. 5, the command bus 124 sends data in addition to instructions.

[0044] The PiM resource manager 108 cooperates with the CPU 104. memory controller 105 and storage controller 107 to dynamically control and manage one or more compute-inmemory (CIM) operations. In some implementations, the CIM operations are executed at the SoC 102 in support of a heterogeneous compute operation between two or more processing units that are included among the IP block 110, the CPU 104, or both. More specifically, the PiM resource manager 108 is configured to generate control signaling via the command 124 and data 125 buses and use one or more discrete signal values of the control signaling to manage and boost data access operations at the memory device 122.

[0045] The system 100 includes an example memory device 122. The memory device 122 can include multiple memory dies, each including memory arrays and integrated compute elements. For example, the memory device 122 can include N memory die. where N is an integer greater than 1 . The memory device 122 can be a dynamic random-access memory (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory' device 122 is configured to perform or support various types of PiM operations, CiM operations, and memory-near-computing operations C’MnC operations”). The memory device 122 performs or supports these operations using its multiple PiM compute elements, which are described below with reference to FIGS. 3-12. The memory arrays may be DRAM or SDRAM arrays. The PiM compute elements (also referred to as integrated compute elements) comprise internal storage such as PiM registers which may also be referred to as PiM register devices or PiM register banks. The internal storage of the PiM compute elements is accessible by the PiM compute elements during execution of instructions by the PiM compute elements operating on data stored in the memory arrays. In an example, the PiM compute elements may execute a multiply-accumulate (MAC) operation using input data stored by the internal storage and weights of a neural network layer stored by the memoryarrays. The memory arrays may have a greater storage capacity than the internal storage of the PiM compute elements.

[0046] The SoC 102 cooperates with the memory device 122 to perform computations across one or more bank groups of the memory device 122. The computations can be for operations or workloads that involve one or more of the processors at IP block 110. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In at least one example the memory device 122 may be external to the SoC 102, whereas in another example the memory device 122 may be internal to the SoC 102.

[0047] In the example of FIG. 1. system 100 and the SoC 102 is an integrated circuit of an example user / client device 130. consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 130a, tablet 130b, laptop 130c, smartwatch or wearable device 130d. The devices 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0048] FIG. 2 is an example prior method 200 of writing data in a PiM architecture using a full command protocol for writing to a memory array. The example method 200 includes a command bus 206, a data bus 208. data to be written 216, and various commands sent via the command bus 206.

[0049] In the prior method of using a full command protocol to write to a memory array, to perform a write operation to a PiM register after closing a previous page 202, a dummy page open command (“active’' or “ACT'’) 210 is sent via the command bus 206. Next, a CAS write command 212 is sent to prepare the receiving memory’ structure (e.g.. DRAM or PiM register) for the write operation. In the case of writing to a PiM register, the CAS write command 212 is followed by a PiM write command 214. The data 216 is then sent via the data bus 208 and written to the PiM register. To conclude the write operation, a dummy page close command (“precharge” or “PRE”) 218 is sent via the command bus 206 to close the dummy page. Other writing operations can then be commenced with another page open command 204. In some examples, the cycle time overhead by issuing both the ACT 210 and PRE 218 command is more than 100 cycles.

[0050] FIG. 3 is an example of improved method 300 of writing data in a PiM architecture (e.g., in the memory device 122 of Fig. 1) using a reduced command protocol for writing to internal storage of the integrated compute elements (e.g., PiM register devices).

[0051] In this improved method 300, commands that are shared with the prior method 200 are indicated with like numbers. The command bus 206 and the data bus 208 are similar to the command bus 124 and the data bus 125 of Fig. 1. In contrast with the prior method 200, the improved method 300 removes the unnecessary ACT 210 and PRE 218 commands. In the improved method 300 for writing data to a PiM register, the command bus 206 first sends the CAS write command 212. In some examples, the CAS write command 212 is modified to indicate to the DRAM that the writing operation is to be conducted to a PiM register. For example, the CAS write command 212 can be modified by designating an unused bit in the command to specify a PiM register write (e.g., “PiM CAS_WR”). An example of unused bits in the CAS write command 212 are the WRITE-X bits (WRX / WXSA), which can be used to specify a PiM register write. By removing the dummy page commands, the cycle time associated with these commands (e.g., 100 cycles) can be eliminated and latency improved.

[0052] FIG. 4, which is a conceptual diagram not considering timing constraints, is an example of improved method 400 of writing to two ranks in a PiM architecture. In this improved method 400. commands that are shared with methods 200 and 300 are indicated with like numbers. In contrast with the previous methods, the improved method 400 depicts a writing operation across multiple PiM register ranks. In this method, commands for two different ranks are issued sequentially on the common command bus 206, with a rank switch timing interval 402 between them. Additionally, the data 216 for rank 0 and data 404 for rank 1 are then placed on the common data bus 208, maintaining the same rank switch interval 402. In this method 400, like method 300, the elimination of dummy page commands saves cycle time and reduces latency.

[0053] FIG. 5 is an example improved method 500 where data is sent via a command bus 206 when the data to be written to the PiM register is below7a threshold size. For example, if the data to be written is 8 bits or smaller, the write command can be sent directly with the command bus 206. Otherwise, if the data is above the threshold size, the data is sent using the data bus 208 as described above in FIGS. 3-4.

[0054] In some examples, to enable writing data using the command bus 206, a command is added to the command bus 206 during idle periods (e.g., a write command for data less than or equal to 8 bits can be sent during a 4-cycle idle period). In some examples, this command can be a 1 -cycle or 2-cycle command depending on the data bits to be transferred. In some examples, this command is a 1-cycle command 502 that is sent prior to a 1-cycle write 504. Additionally, because the command bus can be shared between PiM register ranks,write operations that are sent via the command bus can be broadcasted to multiple ranks simultaneously. In some examples, the host device sends data to all ranks of a multi-rank PiM architecture in parallel (e.g., through the command bus 206). In other examples, the host device sends data to the ranks of the multi -rank PiM architecture in serial (e.g., through the data bus 208).

[0055] FIG. 6 is an example process 600 for implementing the improved method of writing data in a PiM architecture (e.g., the memory device 122 of Fig.1). The PiM architecture comprises one or more memory arrays (e g., DRAM or SDRAM arrays) and one or more integrated compute elements (e.g., PiM compute elements). The one or more integrated compute elements include internal storage (e.g., PiM registers) that is accessible during execution of instructions by the one or more integrated compute elements operating on data stored in the one or more memory arrays. The example process 600 includes receiving data over a data interface using a reduced command protocol (610) and storing the data in the internal storage of the one or more integrated compute elements (620). As described above with respect to FIGS. 3-4, the reduced command protocol eliminates the unnecessary dummy page commands when writing data to a PiM register. The “data interface’7refers to an interface where a data bus (e.g., the data bus 125 or 208) connects to the PiM architecture.

[0056] FIG. 7 is an example process 700 for sending data for writing via a command bus. The example process 700 includes receiving data over a command interface of a processing- in-memory (PIM) subsystem for storage (710). The memory device 122 of Fig. 1 is an example of the PiM subsystem. The PiM subsystem comprises one or more memory arrays (e.g., DRAM or SDRAM arrays) and one or more integrated compute elements (e.g., PiM compute elements). The one or more integrated compute elements include internal storage (e.g., PiM registers) that is accessible during execution of instructions by the one or more integrated compute elements operating on data stored in the one or more memory arrays. The “command interface” refers to an interface where a command bus (e.g., the command bus 124 or 206) connects to the PiM subsystem. The example process 700 further includes storing the data in the internal storage of the one or more integrated compute elements (720). As described with reference to FIG. 5 above, the command bus can be used to send data for writing to the PiM register when the data size is below a threshold.Interleaving of Commands

[0057] Prior approaches to carrying out computations on the PiM architecture involved an input data loading phase followed by a MAC command execution phase. During the inputdata loading phase, the entirety of input data that would be needed by the PiM architecture for during the MAC command execution phase is loaded into an input queue of the memory device. During the MAC command execution phase, the PiM architecture can consume input data from the input queue as each compute command is executed. The input queue, like the PiM register device, can be accessed without accessing the DRAM page.

[0058] In contrast with prior approaches, only a portion of the input data needed by the PiM architecture during the MAC command execution phase is loaded into the input queue during the input data loading phase. The remainder of input data is loaded into the input queue during the MAC command execution phase. In particular, commands for writing input data into the input queue are interleaved with compute or MAC commands during the MAC command execution phase. In this manner, at least a subset of the input data transfer operations of input data loading phase can now occur in parallel with the MAC operations of the MAC command execution phase, rather than entirely sequentially with the execution phase as in some prior approaches.

[0059] FIG. 8 is a block diagram of an example PiM module 800. The PiM module 800 can be used to support process-in-memory operations in the memory device 122 (FIG. 1). The PiM module includes a command decoder 802 coupled with a PiM compute block / module 804 and a read / write module 806. The command decoder 802 can be implemented in hardware, software, or both. The command decoder 802 is communicably coupled with the SoC 102 via the CA bus 124, over which the command decoder 802 can receive compute commands and read / write commands (e.g., input queue write commands) and data from the memory controller 105 or the PiM / CiM resource manager 108.

[0060] The command decoder 802 includes routing logic configured to appropriately route the commands to either the PiM compute module 804 or the read / write module 806. For example, the command decoder 802 can route MAC / compute commands to the PiM compute module 804 and route input queue write commands to the read / write module 806. The PiM module 800 can also include a memory array 812 that can store parameters for computation, where the parameters can include, for example, weights of a neural network layer. The read / write module 806 can carry out read and wnte operations with respect to the memory array 812 as well as the input queue 808.

[0061] In some examples, the read / write module 806 can receive read / write commands from the command decoder 802. An example write command can instruct the read / write module 806 to load / write parameters received from the DQ bus 125 to the memory array 812. In some implementations, the parameters and / or other input data are written to the memoryarray 812 directly from the DQ bus 125 or from the input data buffer 816, which is coupled to the DQ bus 125. The read / write module 806 can also be instructed to load / write input data to the input queue 808 from the input data buffer 816 or directly from the DQ bus 125.Relatedly, a first example read command can instruct the read / write module 806 to read parameters from the memory' array 812 that are then passed to PiM compute module 804, whereas a second example read command can instruct the read / write module 806 to read inputs data values from the input queue 808 that are then passed to PiM compute module 804.

[0062] The PiM compute module 804 can execute, for example, multiply-and-accumulate (MAC) commands, which carry out multiply and accumulate operations based on parameters stored in the memory' array 812 and the input data stored in the input queue 808. The PiM compute module 804 can store or pass the results of the MAC operations in an output queue 810 before writing the results to a memory' array of the memory device 122. In some implementations, the results are passed to another PiM compute module 804 included in the memory' device 122, for computations associated with another neural network layer.

[0063] In some example implementations, during PiM compute operations, the DQ bus 125 may not be accessed. Specifically, the PiM MAC commands utilize parameters stored in the memory array 812 and the input data stored in the input queue 808, but do not access any data on the DQ bus 125. This can allow communication of input data over the DQ bus 125 while the PiM compute module 804 is executing compute operations. In some implementations, the DQ bus 125 can be coupled directly to the read / write module 806, the input queue 808, or both.

[0064] FIG. 9 shows a flow diagram of an example process 900 that can be executed by, or using, the memory' controller 105. The memory' controller 105 can be used to execute one example version of process 900 for providing input data concurrently with MAC operations. For example, the controller 105 can obtain a control value indicating a duration of a compute operation to be executed at the PiM compute module 804. The memory controller 105 can determine, based on the control value, at least one timing parameter for communicating commands to the PiM module. The memory’ controller 105 can execute, based on the timing parameter, an interleaving operation that interleaves a command for writing of input data to the memory device 122 with two or more other ty pes of commands. For example, and as indicated above, each of the two or more other types of commands that are interleaved with the command for writing input data to the memory device 122 can be MAC commands. Further, the memory controller 105 can communicate different types of commands andcorresponding input data to the PiM module based on the interleaving operation. A specific example of this process is shown in FIG. 9 as process 900.

[0065] Another example version of process 900, which may be integrated with the version discussed above, is reflected in the example of Fig. 9, and includes obtaining a control value indicating a duration of a PiM compute command (902). The memory' controller 105 can determine the amount of time taken to execute a PiM compute command. In some examples, the control value indicating this duration can be a static value stored in a register that can be read by the memory controller 105, whereas in some other examples, the control value may be dynamically determined by another module and provided to the memory' controller 105. FIG. 10 shows a timing diagram 1000 of an example sequence of commands provided by the memory controller to the PiM module 800. The timing diagram shows multiple PiM / MAC compute commands 1002 (labeled as ‘‘PIM MAC”) provided by the memory controller 105 onto the CA bus 124. The memory controller can determine the expected duration of execution of the PiM compute commands 1002.

[0066] The process 900 also includes determining, based on the control value, at least one timing parameter for communicating input queue write commands to the PiM module (904). For example, the memory' controller 105 can determine a minimum time duration that indicates when, after MAC operations have been initiated at PiM module, an input queue write command 1004 can be sent or communicated to the PiM module 800. In some instances, the timing parameter represents the minimum time duration and indicates the duration in clock cycles. The timing parameter can also represent a minimum number of clock cycles that sending an input queue write command 1004 can be delayed after sending a previous PiM compute command 1002. In some examples, the timing parameter can be at least one clock cycle and no more than N-l, where N is a minimum number of clock cycles between two successive PiM compute commands 1002. In some implementations, the minimum number of clock cycles between two successive PiM / MAC commands 1002 is dynamically determined for a given workload or task.

[0067] The process 900 includes communicating, to the PiM module, PiM compute commands and input queue write commands that are interleaved with the PiM compute commands based at least on the at least one timing parameter (906). As shown in Figure 10, the memory controller 105 can send PiM compute commands 1002 and input queue write commands 1004 on the command bus to the PiM module 800 such that the input queue write commands are interleaved with the PiM compute commands. The memory controller 105 can utilize the timing parameter to determine the number of clock cycles after the communicationof a PiM compute command to communicate the next interleaved input queue write command 1004.

[0068] Interleaving the input queue write commands 1004 with the PiM compute commands 1006 can reduce the amount of time needed to execute a set of PiM compute commands. In particular unlike prior approaches, where all of the data expected to be consumed by the PiM compute commands is pre-loaded prior to the execution of the PiM commands, the improved technique and approach described in this specification can be used to load input data into the input queue while the PiM compute commands are being executed. This reduces the overhead of loading all of the input data into the input queue. Thus, the total time needed for executing a set of PiM commands can be reduced.

[0069] In some implementations, a command decoder 802 of an example PiM architecture is configured to include independent controls (e.g., control logic) that allow for independent, but concurrent, control of the read / write module 806 and the PiM compute module 804, such that each of modules 804, 806 can execute operations concurrently.

[0070] In some examples, the memory controller 105 can communicate input data for pre-loading to an input / memory queue prior to execution of compute operations at the PiM module. For example, the input queue 808 shown in Figure 8 can be an input (or memory) queue of the memory' device 122, which stores a subset of input data preloaded to the PiM module. In some examples of such communication, the memory controller 105 can communicate one or more input queue write commands 1006 prior to the communication of the PiM compute commands 1002 to preload a predetermined amount of input data in the input queue. The amount of input data pre-loaded into the input queue can be less than the total amount of input data expected to be consumed by the set of PiM compute commands 1002. In some examples, no more than half the total amount of input data needed for executing all commands in the set of PiM compute commands can be pre-loaded into the input queue.

[0071] As an example, assume that 512B of input data is consumed by 64 PiM compute commands, where B = byte and one byte = 8 bits. The memory controller 105 may preload 256B of input data into the input queue. The remaining 256B of input data can be loaded into the input queue by the interleaved input queue write commands 1004 during the execution duration. In some examples, 8 input queue write commands may be needed to load the 256B of input data. In some such instances, the memory controller 105 can communicate 8 input queue write commands to the PiM module for pre-loading 256B of input data and then communicate 8 more input queue write commands 1004 that are interleaved with the PiMcompute commands 1002. Once the 8 interleaved input queue write commands 1004 are communicated, the memory controller 105 can stop sending additional input queue write commands to the PiM module until all of the 64 PiM compute commands have been communicated.

[0072] In some examples, the amount of input data that needs to be pre-loaded can be determined in part on the write latency of the input queue write commands. The greater the latency, the greater the amount of input data that would have to be pre-loaded into the input queue.

[0073] In some examples, the memory’ controller 105 can maintain a counter indicating the available input data in the input data queue and can communicate the input queue write commands based on the value of the counter. The counter can indicate the amount of input data that is present in the input queue. Based on this information, the memory controller 105 can issue communication of the input queue write commands to maintain a predetermined amount of input data in the queue. In some instances, the memory' controller 105, based on the counter indicating insufficient input data present in the input queue can stall the communication of PiM compute commands 1002 and continue sending input queue write commands until the amount of input data in the input queue reaches desirable amounts.

[0074] FIG. 11 shows a block diagram of an example memory' controller 1100. The memory controller 1100 can be used to implement the memory controller 105 discussed herein. The memory controller 1105 can include a scheduler and interleaver module 1102. a counter module 1 1 4, and a timing logic 1106. The scheduler and interleaver module 1 102 can determine the timing with which to memory' controller 1100 can communicate the PiM compute instructions 1002 and the input queue write commands 1004 to the PiM module 800. The counter module 1104 can maintain one or more counters that monitor the amount of input data remaining in the input queue as well as a count of MAC operations that require execution for a given workload or task.

[0075] The timing logic 1106 can determine at least one timing parameter based on a control value indicating the duration of the MAC operations that are triggered for execution by one or more PiM compute commands. The scheduler and interleaver module 1102 can utilize the counter data from the counter module 1104 and the timing information / parameters generated using the timing logic 1106 to determine the timing of communicating PiM compute commands and input queue write commands to memory device 122, such that the input queue write commands are interleaved with the PiM compute commands and the inputwrite operations are interleaved with PiM compute commands in accordance with the timing information determined for a given workload.

[0076] FIG. 12 shows a flow diagram of a process 1200 executed by the PiM compute module. For example, the process 1200 can be executed by the PiM module 800 shown in FIG. 8. In general, the PiM module 800 can execute a process for writing of input data concurrently with execution of PiM commands. The process can include receiving from the controller via a command bus. a write command interleaved with PiM commands. The PiM module 800 can execute the PiM commands to retrieve parameters accessed from a memory array and input data received from the SoC.

[0077] The PiM module 800 can execute a multiply -and-accumulate (MAC) operation using the parameters and input data. Further, the PiM module 800 can, concurrent with executing the MAC operation, execute the write command interleaved with the PiM commands to trigger receipt of additional input data from the SoC. FIG. 12 shows a specific example process 1200 of this general operation. The process 1200 includes receiving, from the memory controller 1100 via a command bus, input queue write commands interleaved with PiM compute commands (1202). Referring to FIG. 8, the command decoder 802 can receive the PiM compute commands and the input queue write commands over the CA bus 124 and send the PiM compute commands to the PiM compute module 804 and send the input queue write commands to the read / write module 806.

[0078] The process also includes executing the PiM compute commands by a PiM compute module based on parameters accessed from a memory array and input data accessed from an input queue (1204). The PiM compute module 804 can execute the PiM compute commands provided by the command decoder 802. The PiM compute commands can include MAC commands that cany’ out multiply-and-accumulate operations. The PiM compute module 804 can access the parameters from the memory array 812 or from a row buffer 814 of the memor array 812. The PiM compute module 804 can access the input data from the input queue 808. The PiM compute module 804 can carry out the MAC operation on the parameters and the input data to generate output data and write the output data to the output data queue 810.

[0079] The process 1200 further includes executing the input queue write commands by a read / write module based on writing input data from a data bus into the input queue, wherein each interleaved input queue w rite command is executed betw een two PiM compute commands (1206). The read / write module 806 can execute the input queue write commands to load input data from the input data buffer 816 into the input queue 808. It should be notedthat the input queue 808 can be configured to allow independent read and write operations. That is. the input queue 808 can allow the PiM compute module 804 to read input data simultaneously while allowing the read / write module 806 to write input data into the input queue 808. The read / write module 806 executes the input queue write commands interleaved with the PiM compute commands. For example, as show n in FIG. 10, the read / write module 806 can execute the input queue write commands 1004 interleaved with the PiM compute commands 1002.

[0080] In some examples, the PiM module 806 can receive from the memory controller 105, prior to receiving the PiM compute commands, a set of input queue write commands. The read / write module 806 can execute this set of input queue write commands to preload input data into the input queue 808. One example of this operation is shown in FIG. 10, where a set of input queue write commands are executed during input data preload duration 1006. After the input data is pre-loaded into the input queue, the PiM compute module 804 can execute at least the first PiM compute command based on the pre-loaded input data.

[0081] In some examples, pre-loading input data into the input queue 808 includes no more than half the amount of data needed for execution by all of the PiM compute commands within an execution cycle. In some examples, a write latency between a time when an input queue write command is executed and a time when the associated input data is loaded into the input queue is greater than the duration that is required to execute the PiM compute command.

[0082] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer softw are or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0083] The term “data processing apparatus'’ refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0084] A computer program which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0085] For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0086] As used in this specification, an “engine.” or “software engine,” refers to a software implemented input / output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs,smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

[0087] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0088] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.

[0089] Generally, a central processing unit will receive instructions and data from a readonly memory7or a random access memory7or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory7devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0090] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory7devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0091] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g, a mouse, trackball, or a presence sensitive display or other surface by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditoryfeedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

[0092] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a frontend component, e.g.. a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0093] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0094] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0095] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0096] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the follow ing claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain cases, multitasking and parallel processing may be advantageous.

[0097] What is claimed is:

Claims

What is claimed is:

1. A method performed by a controller of a system-on-chip (SoC) communicably coupled with a processing-in-memory (PiM) module of a memory device, comprising: obtaining a control value indicating a duration of a compute operation to be executed at the PiM module; determining, based on the control value, at least one timing parameter used to communicate commands to the PiM module, the commands comprising a first command used to write input data to a memory queue coupled to the PiM module and a second command for executing compute operations; executing, by the controller and based on the timing parameter, an interleaving operation that interleaves the first command for writing of input data to the memory queue with two or more of the second command for executing compute operations; and communicating, to the PiM module, the commands and input data based on the interleaving operation.

2. The method of claim 1, further comprising: communicating, to the PiM module, input data for pre-loading to the memory queue prior to execution of the compute operation at the PiM module.

3. The method of claim 2, wherein no more than half the total amount of input data needed for executing all of the compute operations is communicated for pre-loading.

4. The method of any preceding claim, further comprising: maintaining a counter indicating available input data in the memory queue; and communicating, to the PiM module, the input data based on the value of the counter.

5. The method of any preceding claim, wherein the at least one timing parameter indicates a duration between communicating the first command and the second command.

6. A system, comprising: one or more processors; andone or more storage devices storing instructions that when executed by the one or more processors perform operations comprising: obtaining a control value indicating a duration of a compute operation to be executed at the PiM module; determining, based on the control value, at least one timing parameter used to communicate commands to the PiM module, the commands comprising a first command used to write input data to a memory queue coupled to the PiM module and a second command for executing compute operations; executing, by the controller and based on the timing parameter, an interleaving operation that interleaves the first command for writing of input data to the memory queue with two or more of the second command for executing compute operations; and communicating, to the PiM module, the commands and input data based on the interleaving operation.

7. The system of claim 6, the operations further comprising: communicating, to the PiM module, input data for pre-loading to the memory queue prior to execution of the compute operation at the PiM module.

8. The system of claim 7, wherein no more than half the total amount of input data needed for executing all of the compute operations is communicated for pre-loading.

9. The system of any one of claims 6-8. the operations further comprising: maintaining a counter indicating available input data in the memory queue; and communicating, to the PiM module, the input data based on the value of the counter.

10. The system of any one of claims 6-9. wherein the at least one timing parameter indicates a duration between communicating the first command and the second command.

11. A method performed by a processing-in-memory (PiM) module coupled with a controller of a system-on-chip (SoC), comprising:receiving, from the controller via a command bus, an input queue write command interleaved with PiM commands; executing, by the PiM module, the PiM commands to retrieve parameters accessed from a memory’ array and input data received from the SoC; executing, at the PiM module, a multiply-accumulate (MAC) operation using the parameters and input data; and concurrent with executing the MAC operation, executing the input queue write command interleaved with the PiM commands to trigger receipt of additional input data from the SoC.

12. The method of claim 11, further comprising: receiving, from the controller via the command bus prior to receiving the PiM commands, a set of input queue write commands; responsive to receiving the set of input queue write commands, preloading input data received from the SoC; and executing at least a first PiM command of the received PiM commands based on preloaded input data.

13. The method of claim 12, wherein preloading input data includes preloading no more than half an amount of data needed for execution of all of the PiM commands.

14. The method of any one of claims 11 to 13, wherein a write latency between a time when an input queue write command is executed and a time when the associated input data is loaded into an input data queue is greater than a duration of execution of the PiM command.

15. A system, comprising: one or more processors; and one or more storage devices storing instructions that when executed by the one or more processors perform operations comprising: receiving, from the controller via a command bus, an input queue write command interleaved with PiM commands;executing, by the PiM module, the PiM commands to retrieve parameters accessed from a memory’ array and input data received from the SoC; executing, at the PiM module, a multiply -accumulate (MAC) operation using the parameters and input data; and concurrent with executing the MAC operation, executing the input queue write command interleaved with the PiM commands to trigger receipt of additional input data from the SoC.

16. The system of claim 15, the operations further comprising: receiving, from the controller via the command bus prior to receiving the PiM commands, a set of input queue write commands; responsive to receiving the set of input queue write commands, preloading input data received from the SoC; and executing at least a first PiM command of the received PiM commands based on preloaded input data.

17. The system of claim 16, wherein preloading input data includes preloading no more than half an amount of data needed for execution of all of the PiM commands.

18. The system of any one of claims 15-17, wherein a write latency between a time when an input queue write command is executed and a time when the associated input data is loaded into an input data queue is greater than a duration of execution of the PiM command.

Images