Reducing latency in processing in memory operations

Abstract

Methods, systems, and media comprising; a host device configured to execute one or more applications; and a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements, wherein the integrated compute elements comprise internal storage that is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays, wherein the host device is configured to send commands over a command interface to initiate storing data in the one or more memory arrays over a data interface, and wherein the host device is configured to send data for storage in the internal storage of the PIM subsystem over the command interface.

Description

[0001] REDUCING LATENCY IN PROCESSING IN MEMORY OPERATIONS

[0002] BACKGROUND

[0003] This specification generally relates to memory devices used to execute computations.

[0004] Modem computing systems often incorporate a wide variety of compute processing units that each offer different computing capabilities and trade-offs. Efficient execution of a given compute job often involves parsing computations into meaningful sub-tasks or workloads that are mapped to available processor cores of a computing system. The computations may be parsed and mapped based on suitability criteria, such as processor capability, performance, and power. Generally, this overall process of allocating portions of a compute to appropriate processor resources is referred to as heterogeneous compute.

[0005] At least one processor core of the computing system can be an Intellectual Property block (“IP block”) that executes a respective portion of a computational operation for different multimedia workloads. An example use case can involve processing image or speech data captured respectively by a camera or microphone on the mobile device. The SoC can use a heterogeneous compute operation to process input samples derived from the image data, the speech data, or both. An example step in the heterogeneous compute operation can include processing data associated with the input samples using a memory' device that provides in-memory processing or computing capabilities.

[0006] SUMMARY

[0007] This specification describes hardware and software techniques for storing data in a processor-in-memory architecture (“PiM architecture”) used for memory-bound compute workloads. The PiM architecture defines one or more PiM blocks of a memory device and each PiM block includes compute elements, such as a processor unit, mode registers, and one or more arithmetic logic units (ALUs). For example, the PiM block can include discrete processors, processor units, register devices, buffers, etc. that cooperate to form one or more PiM compute elements.

[0008] Prior design approaches for storing data in a PiM architecture require opening a “dummy” page before writing data to a memory' array in a dynamic random access memory DRAM or PiM register device. The dummy page must then be closed once the writing operation is complete. In scenarios where data is to be written to a PiM register device, the steps of opening and closing a dummy page are unnecessary as the DRAM page is not being accessed. In contrast to these prior approaches, the techniques of this specification remove the unnecessary steps of opening and closing dummy pages when writing data to a PiM register device. Additionally, if the size of data being written is below a threshold value, further processing efficiency can be realized by transmitting the data to be written in a command bus of the PiM architecture.

[0009] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Removing the unnecessary' steps of opening and closing dummy pages when writing data to a PiM register device drastically reduces write latency. These techniques also reduce switching latency when conducting writing operations across PiM memory die ranks that share the same command and data buses. In the scenario where the data to be written is small enough to be sent w ith the command bus, these techniques can broadcast the same write command to two ranks simultaneously - thus eliminating switching latency. A reduction in writing latency results in greater overall processing efficiency for the host device.

[0010] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below'. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 is a block diagram of an example computing system with at least one SoC.

[0013] FIG. 2 is an example prior art method of writing data in a PiM architecture.

[0014] FIG. 3 is an example improved method of writing data in a PiM architecture.

[0015] FIG. 4 is an example improved method of writing data to two ranks in a PiM architecture.

[0016] FIG. 5 is an example improved method where data is sent via a command bus.

[0017] FIG. 6 is an example process for implementing the improved method of w riting data in a PiM architecture.

[0018] FIG. 7 is an example process for sending data for writing via a command bus. DETAILED DESCRIPTION

[0019] FIG. 1 is a block diagram of an example computing system 100 that includes a system-on-chip, or host device. 102 (“SoC 102’'). The SoC 102 includes a central processing unit 104 (‘"CPU 104”), a memory controller 105. a shared memory 106 (“memory 106”), a resource manager 108, and an IP / circuit block 1 10. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at system 100. The example computing system can implement the methods and processes described below with respect to FIGS. 2-7.

[0020] The CPU 104 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory' and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0021] The memory 106 is a system memory, shared memory7, or both. In the example of Fig. 1, memory 106 is depicted external to circuit block 110. However, memory7106 can include portions of memory7that are: i) specific to circuit block 110. ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory of the SoC 102, such as static random access memory7(SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0022] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit 1 10. The memory 106 can also include various other ty pes of memory7, such as high bandwidth memory7(HBM), narrow memory7(e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0023] The resource manager 108 is implemented in hardware and software. Aspects of the resource manager 108 can be also implemented as firmware of the SoC 102 or firmware of device of the SoC 102, such as a DRAM memory7device or the CPU 104. The resource manager 108 is a processor-in-memory (PiM) resource manager (“PiM resource manager 108”) that includes control logic implemented in hardware, software, or both. For example, the PiM resource manager 108 can include resources such as flip- flops, registers, buffers, etc. that are implemented in hardware and control logic (e.g.. programmed code) that is implemented in software.

[0024] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116. and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, TPU 114, DSP 116, and GPU 118 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0025] One or more aspects of the PiM resource manager 108 can be implemented as a software routine (or module) of the CPU 104, which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 102, such as memory 106. In some implementations, each processor (e.g., ISP 112, DSP 116, TPU 114, GPU 118) of the SoC 102 includes multiple cores and the CPU 104 and / or the PiM resource manager 108 can generate control signaling via a command bus (“CA BUS”) 124 and data bus (“DQ BUS”) 125 to manage and distribute memory intensive compute operations to a memory device 122 (e.g.. DRAM) to minimize the processing load at each core of the processors. The control signaling of the command 124 and data 125 buses is routed at system 100 using an example bus 120 of the SoC 102. The control signaling of the command 124 and data 125 buses can include commands, requests, data, instructions, or combination of these. In some examples, described below with respect to FIG. 5, the command bus 124 sends data in addition to instructions.

[0026] The PiM resource manager 108 cooperates with the CPU 104, memory controller 105 and storage controller 107 to dynamically control and manage one or more computein-memory (CIM) operations. In some implementations, the CIM operations are executed at the SoC 102 in support of a heterogeneous compute operation between two or more processing units that are included among the IP block 110, the CPU 104, or both. More specifically, the PiM resource manager 108 is configured to generate control signaling via the command 124 and data 125 buses and use one or more discrete signal values of the control signaling to manage and boost data access operations at the memory device 122.

[0027] The system 100 includes an example memory device 122. The memory device 122 can include multiple memory dies, each including memory' arrays and integrated compute elements. For example, the memory device 122 can include N memory die, where N is an integer greater than 1. The memory device 122 can be a dynamic randomaccess memory (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory device 122 is configured to perform or support various types of PiM operations, CiM operations, and memory-near-computing operations ("MnC operations'’). The memory device 122 performs or supports these operations using its multiple PiM compute elements, which are described below with reference to FIGS. 3-7. The memory arrays can be DRAM or SDRAM arrays. The PiM compute elements (also referred to as integrated compute elements) comprise internal storage such as PiM registers which can also be referred to as PiM register devices or PiM register banks. The internal storage of the PiM compute elements is accessible by the PiM compute elements during execution of instructions by the PiM compute elements operating on data stored in the memory arrays. In an example, the PiM compute elements execute a multiply-accumulate (MAC) operation using input data stored by the internal storage and weights of a neural network layer stored by the memory' arrays. The memory arrays can have a greater storage capacity than the internal storage of the PiM compute elements.

[0028] The SoC 102 cooperates with the memotv device 122 to perform computations across one or more bank groups of the memory device 122. The computations can be for operations or workloads that involve one or more of the processors at IP block 110. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In at least one example the memory' device 122 may be external to the SoC 102, whereas in another example the memory’ device 122 may be internal to the SoC 102.

[0029] In the example of FIG. 1. system 100 and the SoC 102 is an integrated circuit of an example user / client device 130, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 130a, tablet 130b, laptop 130c, smartwatch or wearable device 130d. The devices 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0030] FIG. 2 is an example prior art method 200 of writing data in a PiM architecture using a full command protocol for writing to a memory array. The example method 200 includes a command bus 206, a data bus 208, data to be written 216, and various commands sent via the command bus 206. In the prior art method of using a full command protocol to write to a memory’ array, to perform a write operation to a PiM register after closing a previous page 202, a dummy page open command (“active” or “ACT”) 210 is sent via the command bus 206. Next, a CAS write command 212 is sent to prepare the receiving memory structure (e.g., DRAM or PiM register) for the write operation. In the case of writing to a PiM register, the CAS write command 212 is followed by a PiM write command 214. The data 216 is then sent via the data bus 208 and written to the PiM register. To conclude the write operation, a dummy page close command (“precharge” or “PRE”) 218 is sent via the command bus 206 to close the dummy page. Other writing operations can then be commenced with another page open command 204. In some examples, the cycle time overhead by issuing both the ACT 210 and PRE 218 command is about 100 cycles.

[0031] FIG. 3 is an example of improved method 300 of writing data in a PiM architecture (e.g., the memory device 122 of Fig. 1) using a reduced command protocol for writing to internal storage of the integrated compute elements (e.g., PiM register devices). In this improved method 300, commands that are shared with the prior art method 200 are indicated with like numbers. The command bus 206 and the data bus 208 are similar to the command bus 124 and the data bus 125 of Fig. 1. In contrast with the prior art method 200, the improved method 300 removes the unnecessary ACT 210 and PRE 218 commands. In the improved method 300 for writing data to a PiM register, the command bus 206 first sends the CAS write command 212. In some examples, the CAS write command 212 is modified to indicate to the DRAM that the writing operation is to be conducted to a PiM register. For example, the CAS write command 212 can be modified by designating an unused bit in the command to specify a PiM register write (e.g., “PiM CAS_WR”). An example of unused bits in the CAS write command 212 are the WRITE-X bits (WRX / WXSA), which can be used to specify a PiM register write. By removing the dummy page commands, the cycle time associated with these commands (e.g., 100 cycles) can be eliminated and latency improved.

[0032] FIG. 4 is an example of improved method 400 of writing to two ranks in a PiM architecture. In this improved method 400, commands that are shared with methods 200 and 300 are indicated with like numbers. In contrast with the previous methods, the improved method 400 depicts a writing operation across multiple PiM register ranks. In this method, commands for multiple ranks are issued concurrently on the common command bus 206 (e.g., PiM write command 214 for rank “0” is issued at the same time the PiM CAS write command 212 is issued for rank “1.” Additionally, a rank switch command 402 is issued by the command bus 206 after the data 216 has been written for the preceding bank in the multi-bank writing operation (e.g., once data 216 has finished being written to rank 0, the rank switch command 402 is issued prior to writing data 404 to rank 1). In this method 400, like method 300, the elimination of dummy page commands saves cycle time and reduces latency.

[0033] FIG. 5 is an example improved method 500 where data is sent via a command bus 206 when the data to be written to the PiM register is below a threshold size. For example, if the data to be written is 8 bits or smaller, the write command can be sent directly with the command bus 206. Otherwise, if the data is above the threshold size, the data is sent using the data bus 208 as described above in FIGS. 3-4. In some examples, to enable writing data using the command bus 206, a command is added to the command bus 206 during idle periods (e.g., a write command for data less than or equal to 8 bits can be sent during a 4-cycle idle period). In some examples, this command can be a 1- cycle or 2-cycle command depending on the data bits to be transferred. In some examples, this command is a 1 -cycle command 502 that is sent prior to a 1 -cycle write 504. Additionally, because the command bus can be shared between PiM register ranks, write operations that are sent via the command bus can be broadcasted to multiple ranks simultaneously. In some examples, the host device sends data to all ranks of a multi-rank PiM architecture in parallel (e g., through the command bus 206). In other examples, the host device sends data to the ranks of the multi-rank PiM architecture in serial (e.g., through the data bus 208).

[0034] FIG. 6 is an example process 600 for implementing the improved method of writing data in a PiM architecture (e.g., the memory device 122 of Fig.l). The PiM architecture comprises one or more memory arrays (e.g., DRAM or SDRAM arrays) and one or more integrated compute elements (e.g., PiM compute elements). In some examples, the one or more memory arrays has a higher storage capacity than the internal storage of the integrated compute elements. The one or more integrated compute elements include internal storage (e.g., PiM registers) that is accessible during execution of instructions by the one or more integrated compute elements operating on data stored in the one or more memory arrays. The example process 600 includes receiving data over a data interface using a reduced command protocol (610) and storing the data in the internal storage of the one or more integrated compute elements (620). As described above with respect to FIGS. 3-4, the reduced command protocol eliminates the unnecessary' dummy page commands when writing data to a PiM register. The “data interface" refers to an interface where a data bus (e.g., the data bus 125 or 208) connects to the PiM architecture. In some examples, the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory' arrays for reading. In some examples, the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays. In some examples, the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memory arrays for reading. In some examples, the host device is configured to finish writing data to the internal storage before operations for opening the portion of the one or more memory arrays for reading has completed

[0035] FIG. 7 is an example process 700 for sending data for writing via a command bus. The example process 700 includes receiving data over a command interface of a processing-in-memory (PiM) subsystem for storage (710). The memory device 122 of Fig.l is an example of the PiM subsystem. The PiM subsystem comprises one or more memory’ arrays (e.g., DRAM or SDRAM arrays) and one or more integrated compute elements (e.g., PiM compute elements). The one or more integrated compute elements include internal storage (e.g., PiM registers) that is accessible during execution of instructions by the one or more integrated compute elements operating on data stored in the one or more memory arrays. The “command interface'’ refers to an interface where a command bus (e.g.. the command bus 124 or 206) connects to the PiM subsystem. The example process 700 further includes storing the data in the internal storage of the one or more integrated compute elements (720). As described in FIG. 5 above, the command bus is used to send data for writing to the PiM register when the data size is below a threshold.

[0036] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of. data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially- generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0037] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g.. files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0038] For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0039] As used in this specification, an “engine,” or “software engine,” refers to a software implemented input / output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory7or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry7. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g.. a universal serial bus (USB) flash drive, to name just a few.

[0040] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory7devices, e.g., EPROM, EEPROM, and flash memory7devices; magnetic disks, e.g., internal hard disks or removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks.

[0041] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g.. a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g. a mouse, trackball, or a presence sensitive display or other surface by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory7feedback, e.g.. visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return.

[0042] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end. middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0043] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0044] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0045] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0046] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain cases, multitasking and parallel processing may be advantageous.

[0047] In addition to the embodiments described above, the following embodiments are also innovative:

[0048] Embodiment 1 is a system comprising: a host device configured to execute one or more applications; and a processing-in-memory (PIM) subsystem comprising one or more memory' arrays and one or more integrated compute elements, wherein the integrated compute elements comprise internal storage that is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays, and wherein the host device is configured to send data over a data interface using a reduced command protocol and to store data in the internal storage of the integrated compute elements.

[0049] Embodiment 2 is the system of embodiment 1, further comprising a full command protocol that includes a first command for opening a portion of the one or more memory arrays, and wherein the reduced command protocol omits the first command.

[0050] In some examples, the system can be configured to send further data over the data interface using the full command protocol and to store the further data in the one or more memory’ arrays.

[0051] In some examples, the first command can be for opening a portion of a receiving memory' structure

[0052] Embodiment 3 is the system of embodiment 2, wherein the full command protocol includes a second command for closing a portion of the one or more memory arrays, and wherein the reduced command protocol omits the second command.

[0053] In some examples, the second command can be for closing a portion of the receiving memory' structure. Embodiment 4 is the system of any one of embodiments 1-3, wherein the reduced command protocol includes a modified write command that indicates that the host device is storing data in the internal storage of the integrated compute elements.

[0054] Embodiment 5 is the system of any one of embodiments 1 -4, wherein the host device is configured to use the one or more memory arrays of the PIM subsystem as a main memory.

[0055] Embodiment 6 is the system of any one of embodiments 2-3, wherein the host device is configured to send commands to the PIM subsystem using a command interface.

[0056] Embodiment 7 is the system of embodiment 6, wherein the system is configured to use the command interface for the full command protocol and the reduced command protocol.

[0057] In some examples, system is configured to use the command interface to send the commands to the PIM subsystem for the full command protocol and the reduced command protocol.

[0058] Embodiment 8 is the system of any one of embodiments 1-7, wherein the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory arrays for reading.

[0059] Embodiment 9 is the system of embodiment 8, wherein the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays.

[0060] Embodiment 10 is the system of any one of embodiments 8-9, wherein the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memory arrays for reading.

[0061] Embodiment 11 is the system of any one of embodiments 9-10, wherein the PIM subsystem is a multi-rank subsystem, and wherein the host device is configured to perform operations comprising: performing a first reduced command protocol for a first rank; performing a second reduced command protocol for a second rank; writing first data to first internal storage of the first rank; issuing a rank switch command after writing data to the internal storage; and writing second data to second internal storage of the second rank.

[0062] Embodiment 12 is the system of embodiment 11, wherein the first reduced command protocol and the second reduced command protocol are at least partially overlapping.

[0063] Embodiment 13 is the system of any one of embodiments 11-12, wherein the second reduced command protocol is at least partially overlapping with writing the first data to the first internal storage of the first rank Embodiment 14 is the system of any one of embodiments 11-13, wherein the system is configured to finish writing the second data to the second internal storage before operations for opening the portion of the one or more memory arrays for reading has completed.

[0064] Embodiment 15 is a method performed by a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements with internal storage, the method comprising: receiving data over a data interface from a host device for storage in the internal storage; and storing the data in the internal storage, wherein the data is sent from the host device using a reduced command protocol, and wherein the internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

[0065] Embodiment 16 is the method of embodiment 15, further comprising a full command protocol that includes a first command for opening a portion of the one or more memory' arrays, and wherein the reduced command protocol omits the first command.

[0066] Embodiment 17 is the method of embodiment 16, wherein the full command protocol includes a second command for closing a portion of the one or more memory arrays, and wherein the reduced command protocol omits the second command.

[0067] Embodiment 18 is the method of any one of embodiments 15-17, wherein the reduced command protocol includes a modified write command that indicates that the host device is storing data in the internal storage of the integrated compute elements.

[0068] Embodiment 19 is the method of any one of embodiments 15-18, wherein the host device is configured to use the one or more memory' arrays of the PIM subsystem as a main memory.

[0069] Embodiment 20 is the method of any one of embodiments 16-17, wherein the host device is configured to send commands to the PIM subsystem using a command interface.

[0070] Embodiment 21 is the method of embodiment 20, wherein the command interface is used for the full command protocol and the reduced command protocol.

[0071] Embodiment 22 is the method of any one of embodiments 15-21, wherein the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory arrays for reading.

[0072] Embodiment 23 is the method of embodiment 22, wherein the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays.

[0073] Embodiment 24 is the method of any one of embodiments 22-23, wherein the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memory arrays for reading. Embodiment 25 is the method of any one of embodiments 23-24, wherein the PIM subsystem is a multi-rank subsystem, and wherein the host device is configured to perform operations comprising: performing a first reduced command protocol for a first rank; performing a second reduced command protocol for a second rank; writing first data to first internal storage of the first rank; issuing a rank switch command after writing data to the internal storage; and writing second data to second internal storage of the second rank.

[0074] Embodiment 26 is the method of embodiment 25, wherein the first reduced command protocol and the second reduced command protocol are at least partially overlapping.

[0075] Embodiment 27 is the method of any one of embodiments 25-26, wherein the second reduced command protocol is at least partially overlapping with writing the first data to the first internal storage of the first rank.

[0076] Embodiment 28 is the method of any one of embodiments 25-27, wherein writing the second data to the second internal storage is finished before operations for opening the portion of the one or more memory arrays for reading has completed.

[0077] Embodiment 29 is one or more non-transi lory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving data over a data interface from a host device for storage in internal storage of one or more integrated compute elements of a processing-in-memory (PIM) subsystem comprising one or more memory' arrays; and storing the data in the internal storage, wherein the data is sent from the host device using a reduced command protocol, and wherein the internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

[0078] Embodiment 30 is the media of embodiment 29, further comprising a full command protocol that includes a first command for opening a portion of the one or more memory’ arrays, and wherein the reduced command protocol omits the first command.

[0079] Embodiment 31 is the media of embodiment 30, wherein the full command protocol includes a second command for closing a portion of the one or more memory arrays, and wherein the reduced command protocol omits the second command.

[0080] Embodiment 32 is the media of any one of embodiments 29-31 , wherein the reduced command protocol includes a modified write command that indicates that the host device is storing data in the internal storage of the integrated compute elements. Embodiment 33 is the media of any one of embodiments 29-32, wherein the host device is configured to use the one or more memory arrays of the PIM subsystem as a main memory.

[0081] Embodiment 34 is the media of any one of embodiments 30-31 , wherein the host device is configured to send commands to the PIM subsystem using a command interface.

[0082] Embodiment 35 is the media of embodiment 34, wherein the command interface is used for the full command protocol and the reduced command protocol.

[0083] Embodiment 36 is the media of any one of embodiments 29-35, wherein the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory arrays for reading.

[0084] Embodiment 37 is the media of embodiment 36, wherein the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays.

[0085] Embodiment 38 is the media of any one of embodiments 36-37, wherein the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memon arrays for reading.

[0086] Embodiment 39 is the media of any one of embodiments 37-38, wherein the PIM subsystem is a multi-rank subsystem, and wherein the host device is configured to perform operations comprising: performing a first reduced command protocol for a first rank; performing a second reduced command protocol for a second rank; writing first data to first internal storage of the first rank; issuing a rank switch command after writing data to the internal storage; and writing second data to second internal storage of the second rank.

[0087] Embodiment 40 is the media of embodiment 39, wherein the first reduced command protocol and the second reduced command protocol are at least partially overlapping.

[0088] Embodiment 41 is the media of any one of embodiments 39-40, wherein the second reduced command protocol is at least partially overlapping with writing the first data to the first internal storage of the first rank.

[0089] Embodiment 42 is the media of any one of embodiments 39-41, wherein writing the second data to the second internal storage is finished before operations for opening the portion of the one or more memory arrays for reading has completed.

[0090] Embodiment 43 is a system comprising: a host device configured to execute one or more applications; and a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements, wherein the integrated compute elements comprise internal storage that is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays, wherein the host device is configured to send commands over a command interface to initiate storing data in the one or more memory arrays over a data interface, and wherein the host device is configured to send data for storage in the internal storage of the PIM subsystem over the command interface.

[0091] Embodiment 44 is the system of embodiment 43, wherein the host device is configured to use the command interface to send data for storage in the internal storage when the size of the data is below a threshold.

[0092] Embodiment 45 is the system of embodiment 44, wherein the host device is configured to use the data interface to send data for storage in the internal storage when the size of the data is equal to or above the threshold.

[0093] Embodiment 46 is the system of any one of embodiments 43-45, wherein the PIM subsystem is a multi-rank system, and wherein the system is configured to send data over the command interface to all ranks of the multi-rank system at least partially in parallel.

[0094] Embodiment 47 is the system of embodiment 46, wherein the host device is configured to send data over the data interface to the ranks of the multi-rank system in serial.

[0095] Embodiment 48 is a method performed by a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements with internal storage, the method comprising: receiving data over a command interface from a host device for storage; and storing the data in the internal storage, wherein the internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

[0096] Embodiment 49 is the method of embodiment 48, wherein the host device is configured to use the command interface to send data for storage in the internal storage when the size of the data is below a threshold.

[0097] Embodiment 50 is the method of embodiment 49, wherein the host device is configured to use the data interface to send data for storage in the internal storage when the size of the data is equal to or above the threshold.

[0098] Embodiment 51 is the method of any one of embodiments 48-50, wherein the PIM subsystem is a multi-rank system, and wherein the system is configured to send data over the command interface to all ranks of the multi-rank system at least partially in parallel.

[0099] Embodiment 52 is the method of embodiment 51, wherein the host device is configured to send data over the data interface to the ranks of the multi-rank system in serial. Embodiment 53 is one or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving data over a command interface from a host device for storage in internal storage of one or more integrated compute elements of a processing-in-memory (PIM) subsystem comprising one or more memory arrays; and storing the data in the internal storage, wherein internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

[0100] Embodiment 54 is the media of embodiment 53, wherein the host device is configured to use the command interface to send data for storage in the internal storage when the size of the data is below a threshold.

[0101] Embodiment 55 The media of embodiment 54, wherein the host device is configured to use the data interface to send data for storage in the internal storage when the size of the data is equal to or above the threshold.

[0102] Embodiment 56 is the media of any one of embodiments 53-55, wherein the PIM subsystem is a multi-rank system, and wherein the system is configured to send data over the command interface to all ranks of the multi-rank system at least partially in parallel.

[0103] Embodiment 57 is the media of embodiment 56, wherein the host device is configured to send data over the data interface to the ranks of the multi-rank system in serial.

[0104] What is claimed is:

Claims

CLAIMS1. A system comprising: a host device configured to execute one or more applications; and a processing- in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements, wherein the integrated compute elements comprise internal storage that is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays, wherein the host device is configured to send data over a data interface using a reduced command protocol and to store data in the internal storage of the integrated compute elements.

2. The system of claim 1, further comprising a full command protocol that includes a first command for opening a portion of the one or more memory arrays, and wherein the reduced command protocol omits the first command.

3. The system of claim 2. wherein the full command protocol includes a second command for closing a portion of the one or more memory arrays, and wherein the reduced command protocol omits the second command.

4. The system of any one of claims 1-3, wherein the reduced command protocol includes a modified write command that indicates that the host device is storing data in the internal storage of the integrated compute elements.

5. The system of any one of claims 1-4, wherein the host device is configured to use the one or more memory arrays of the PIM subsystem as a main memory-.

6. The system of any one of claims 2-3, wherein the host device is configured to send commands to the PIM subsystem using a command interface.

7. The system of claim 6, wherein the system is configured to use the command interface for the full command protocol and the reduced command protocol.

8. The system of any one of claims 1-7, wherein the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory arrays for reading.

9. The system of claim 8, wherein the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays.

10. The system of any one of claims 8-9, wherein the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memory arrays for reading.

11. The system of any one of claims 9-10, wherein the PIM subsystem is a multi-rank subsystem, and wherein the host device is configured to perform operations comprising: performing a first reduced command protocol for a first rank; performing a second reduced command protocol for a second rank; writing first data to first internal storage of the first rank; issuing a rank switch command after writing data to the internal storage; and writing second data to second internal storage of the second rank.

12. The system of claim 11, wherein the first reduced command protocol and the second reduced command protocol are at least partially overlapping.

13. The system of any one of claims 11-12, wherein the second reduced command protocol is at least partially overlapping with writing the first data to the first internal storage of the first rank.

14. The system of any one of claims 11-13, wherein the system is configured to finish writing the second data to the second internal storage before operations for opening the portion of the one or more memory arrays for reading has completed.

15. A method performed by a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements with internal storage, the method comprising: receiving data over a data interface from a host device for storage in the internal storage; andstoring the data in the internal storage, wherein the data is sent from the host device using a reduced command protocol, and wherein the internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

16. The method of claim 15, further comprising a full command protocol that includes a first command for opening a portion of the one or more memory arrays, and wherein the reduced command protocol omits the first command.

17. The method of claim 16, wherein the full command protocol includes a second command for closing a portion of the one or more memory arrays, and wherein the reduced command protocol omits the second command.

18. The method of any one of claims 15-17, wherein the reduced command protocol includes a modified write command that indicates that the host device is storing data in the internal storage of the integrated compute elements.

19. The method of any one of claims 15-18, wherein the host device is configured to use the one or more memory arrays of the PIM subsystem as a main memory.

20. The method of any one of claims 16-17, wherein the host device is configured to send commands to the PIM subsystem using a command interface.

21. The method of claim 20, wherein the command interface is used for the full command protocol and the reduced command protocol.

22. The method of any one of claims 15-21, wherein the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory' arrays for reading.

23. The method of claim 22, wherein the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays.

24. The method of any one of claims 22-23, wherein the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memoiy arrays for reading.

25. The method of any one of claims 23-24, wherein the PIM subsystem is a multirank subsystem, and wherein the host device is configured to perform operations comprising: performing a first reduced command protocol for a first rank; performing a second reduced command protocol for a second rank; writing first data to first internal storage of the first rank; issuing a rank switch command after writing data to the internal storage; and writing second data to second internal storage of the second rank.

26. The method of claim 25, wherein the first reduced command protocol and the second reduced command protocol are at least partially overlapping.

27. The method of any one of claims 25-26, wherein the second reduced command protocol is at least partially overlapping with writing the first data to the first internal storage of the first rank.

28. The method of any one of claims 25-27, wherein writing the second data to the second internal storage is finished before operations for opening the portion of the one or more memory arrays for reading has completed.

29. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving data over a data interface from a host device for storage in internal storage of one or more integrated compute elements of a processing-in-memory (PIM) subsystem comprising one or more memoiy arrays; and storing the data in the internal storage, wherein the data is sent from the host device using a reduced command protocol, and wherein the internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

30. The media of claim 29, further comprising a full command protocol that includes a first command for opening a portion of the one or more memory arrays, and wherein the reduced command protocol omits the first command.

31. The media of claim 30, wherein the full command protocol includes a second command for closing a portion of the one or more memory arrays, and wherein the reduced command protocol omits the second command.

32. The media of any one of claims 29-31, wherein the reduced command protocol includes a modified write command that indicates that the host device is storing data in the internal storage of the integrated compute elements.

33. The media of any one of claims 29-32, wherein the host device is configured to use the one or more memory arrays of the PIM subsystem as a main memory.

34. The media of any one of claims 30-31, wherein the host device is configured to send commands to the PIM subsystem using a command interface.

35. The media of claim 34, wherein the command interface is used for the full command protocol and the reduced command protocol.

36. The media of any one of claims 29-35, wherein the host device is configured to overlap the reduced command protocol with operations for opening a portion of the one or more memory arrays for reading.

37. The media of claim 36, wherein the host device is configured to overlap writing to the internal storage with the operations for opening the portion of the one or more memory arrays.

38. The media of any one of claims 36-37, wherein the reduced command protocol and writing to the internal storage occupies fewer clock cycles than opening the portion of the one or more memory arrays for reading.

39. The media of any one of claims 37-38, wherein the PIM subsystem is a multi-rank subsystem, and wherein the host device is configured to perform operations comprising:performing a first reduced command protocol for a first rank; performing a second reduced command protocol for a second rank; writing first data to first internal storage of the first rank; issuing a rank switch command after writing data to the internal storage; and writing second data to second internal storage of the second rank.

40. The media of claim 39, wherein the first reduced command protocol and the second reduced command protocol are at least partially overlapping.

41. The media of any one of claims 39-40, wherein the second reduced command protocol is at least partially overlapping with writing the first data to the first internal storage of the first rank.

42. The media of any one of claims 39-41, wherein writing the second data to the second internal storage is finished before operations for opening the portion of the one or more memory arrays for reading has completed.

43. A system comprising: a host device configured to execute one or more applications; and a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements. wherein the integrated compute elements comprise internal storage that is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays, wherein the host device is configured to send commands over a command interface to initiate storing data in the one or more memory arrays over a data interface, and wherein the host device is configured to send data for storage in the internal storage of the PIM subsystem over the command interface.

44. The system of claim 43. wherein the host device is configured to use the command interface to send data for storage in the internal storage when the size of the data is below a threshold.

45. The system of claim 44, wherein the host device is configured to use the data interface to send data for storage in the internal storage when the size of the data is equal to or above the threshold.

46. The system of any one of claims 43-45, wherein the PIM subsystem is a multirank system, and wherein the system is configured to send data over the command interface to all ranks of the multi-rank system at least partially in parallel.

47. The system of claim 46, wherein the host device is configured to send data over the data interface to the ranks of the multi-rank system in serial.

48. A method performed by a processing-in-memory (PIM) subsystem comprising one or more memory arrays and one or more integrated compute elements with internal storage, the method comprising: receiving data over a command interface from a host device for storage; and storing the data in the internal storage, wherein the internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory arrays.

49. The method of claim 48, wherein the host device is configured to use the command interface to send data for storage in the internal storage when the size of the data is below a threshold.

50. The method of claim 49, wherein the host device is configured to use the data interface to send data for storage in the internal storage when the size of the data is equal to or above the threshold.

51. The method of any one of claims 48-50, wherein the PIM subsystem is a multirank system, and wherein the system is configured to send data over the command interface to all ranks of the multi-rank system at least partially in parallel.

52. The method of claim 51, wherein the host device is configured to send data over the data interface to the ranks of the multi-rank system in serial.

53. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving data over a command interface from a host device for storage in internal storage of one or more integrated compute elements of a processing-in- memory (PIM) subsystem comprising one or more memory7arrays; and storing the data in the internal storage, wherein internal storage is accessible during execution of instructions by the PIM subsystem operating on data stored in the one or more memory' arrays.

54. The media of claim 53, wherein the host device is configured to use the command interface to send data for storage in the internal storage when the size of the data is below a threshold.

55. The media of claim 54, wherein the host device is configured to use the data interface to send data for storage in the internal storage when the size of the data is equal to or above the threshold.

56. The media of any one of claims 53-55, wherein the PIM subsystem is a multirank system, and wherein the system is configured to send data over the command interface to all ranks of the multi-rank system at least partially in parallel.

57. The media of claim 56, wherein the host device is configured to send data over the data interface to the ranks of the multi-rank system in serial.

Images