In-memory processing architecture for memory-constrained computing workloads

Abstract

Methods and systems for implementing a Process-in-Memory (PiM) architecture in an integrated memory device such as a DRAM device are described, including a computer-readable medium. The PiM architecture is operable to perform memory-constrained and computationally-constrained machine learning and / or neural network inference (or training) workloads. Techniques for implementing the following are also described: i) a debugging process that uses the computational elements of the PiM architecture to determine the state of computation performed in the memory device; ii) an access control process that restricts the range of memory addresses and locations of the memory device that the PiM architecture is allowed to access; and iii) a data refresh process that optimizes when and how refresh signals are issued across groups and rows of memory cells in the memory device.

Description

Technical Field

[0001] This specification generally relates to an architecture for performing computations in a memory device coupled to a system-on-a-chip (“SoC”). Background Technology

[0002] Modern computing systems typically incorporate various computing processing units, each offering different computing capabilities and trade-offs. Efficient execution of a given computational job usually involves parsing the computation into meaningful workloads or subtasks, which are then mapped to the available processor cores of the computing system. Computation can be parsed and mapped based on suitability criteria such as processor capacity, performance, and power consumption. This overall process of allocating different parts of the computation to appropriate processor resources is often referred to as heterogeneous computing.

[0003] At least one processing unit of the computing system may be an intellectual property block (“IP block”) that performs a corresponding portion of a computational operation for different multimedia use cases. Example use cases may involve processing image or audio data captured by a camera or microphone on a mobile device, respectively. The SoC may use heterogeneous computing operations to process input samples derived from image data, audio data, or both. Example steps in a heterogeneous computing operation may include providing input samples from image data to the SoC’s host processor (such as a neural network processor or machine learning (ML) engine) to generate inference output. Summary of the Invention

[0004] This specification describes techniques for implementing debugging and instruction / command monitoring, access control procedures, and data refresh operations in a memory-in-processing architecture (“PiM architecture”) for memory-constrained and compute-constrained workloads executed in a memory device. These techniques include generating data and control signaling at a system-on-a-chip (SoC) that communicates with the PiM architecture via the memory device. The data and control signaling are routed to and processed by the compute elements of the PiM architecture to trigger the execution of certain data processing and compute operations within the memory device.

[0005] PiM architectures may include discrete processors, processor units, register devices, buffers, etc. PiM architectures are integrated into example memory devices, such as Dynamic Random Access Memory (DRAM) or Double Data Rate (DDR) Synchronous DRAM (SDRAM). Memory devices may be external to (or internal to) the SoC and configured to perform in-memory processing operations and in-memory computation operations (“CiM operations”) using their multiple PiM computing elements. Memory devices collaborate with the SoC to perform computations across one or more memory dies of the memory device. For example, the PiM architecture within the memory device configures, triggers, and executes computations within the PiM architecture's computing units based on data and control signaling generated at the SoC.

[0006] Debugging: Regarding debugging and instruction monitoring, the SoC uses the disclosed techniques to implement and perform debugging processes to determine the status of computations and related operations performed by PiM architecture computing elements in the memory device. Performing the debugging process includes identifying and mitigating errors that occur during processing (or computation) operations in one or more memories. The SoC may trigger the execution of computational operations in the memory device based on request (or command) signals generated by the SoC's memory controller, for example, using the PiM architecture. Request signals are passed to PiM architecture computing elements in the memory device to trigger the execution of instructions in the PiM instruction buffer. For example, request signals may be passed to the PiM architecture processor, the PiM architecture mode register, or both.

[0007] The processor uses request signals to execute computation instructions and generates response signals that indicate the status of the operation performed in response to the request signal. These operations include ML and / or neural network computations involving data (or operands) such as inputs and weights / parameters, stored in a memory bank of a memory device, transferred from the SoC to the memory device, or both. The status can indicate the success or failure of discrete steps or instructions in the processing (or computation) operation used for a request-based ML or neural network inference task or workload. For example, the response signal indicates whether an instruction has been successfully executed or whether an error has occurred indicating that the requested action failed. In some implementations, PiM architecture elements or devices generate flags that are passed to the SoC to indicate the success or failure of an instruction.

[0008] Access Control: Regarding access control, the SoC uses the disclosed techniques to implement and execute access control procedures that restrict the range of memory addresses and memory cell locations of the memory device that the PiM architecture is allowed to access. The SoC determines the memory access range and generates a command that transmits the memory access range to the memory device. This command is used to configure memory access control at the memory device based on the memory access range specified by the command. For example, the command can trigger or configure access controls that restrict PiM computing elements to access only specific memory portions within the memory device.

[0009] Commands are part of an access control procedure used to restrict the ability of computing elements in a PiM architecture to access or modify data at specific locations on a memory device. For example, an access control procedure can restrict a mode register or processing unit in a PiM architecture to access only specific rows and / or groups of memory devices that contain the data and instructions required for computational operations assigned to a specific memory die. Commands can define (e.g., "carve out") specific rows and / or groups of memory using the physical address range of adjacent regions of the memory device.

[0010] Refresh Operations: Regarding refresh optimization, the SoC uses the disclosed techniques to implement and execute the data refresh process to optimize when and how refresh signals are issued across rows of memory cells in a memory device (such as DRAM or associated integrated memory devices). The memory controller on the SoC can be configured to issue a threshold number of refresh commands to ensure compliance with refresh requirements associated with the memory device. For example, refresh requirements for the memory device may specify a minimum refresh interval at which memory cells of the memory device need to receive a refresh (e.g., a refresh signal). Refresh operations are performed to maintain the integrity of the data stored in the memory cells at each row of the memory device.

[0011] The memory controller can be integrated on the SoC and is operable to interact with PiM architecture components to manage and monitor refresh operations initiated at the memory device. The memory controller is configured to optimize the execution of refresh operations on memory banks of the memory device by referencing ML and other computational operations performed at the PiM architecture of the memory device. In some implementations, the memory controller optimizes the timing, ordering, and / or scheduling of when refresh operations are performed at the memory device. For example, memory controller 105 optimizes the timing and scheduling of refreshes to minimize interruptions to ML inference computations performed by the PiM architecture using data obtained from memory banks that require periodic refreshes.

[0012] Different elements and processing units within the PiM architecture are used to optimize the refresh process in the memory device. For example, computational elements such as the mode register and processing unit of the PiM architecture are used to actively (or dynamically) trigger the refresh of memory cells at each of the rows of the memory device. More specifically, the refresh process is optimized by actively (or dynamically) triggering the refresh of memory cells at a frequency exceeding the minimum refresh requirements for refreshing rows of the memory device. For example, instead of based on a predefined refresh rate as described in standard documentation or by the vendor, the internal controller of the PiM architecture can actively (or dynamically) perform refreshes based on the frequency of a clock cycle generated by the memory controller.

[0013] One aspect of the subject matter described in this specification can be embodied in a computer-implemented method, the method comprising generating a first signal at a system-on-a-chip (SoC); and triggering the execution of a computational operation in a memory device coupled to the SoC based on the first signal. The method further comprises, after triggering the execution of the computational operation in the memory device, receiving from the memory device a second signal generated in or referenced to a memory die of the memory device; and determining a state of the computational operation based on the second signal.

[0014] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the first signal is a read request signal that uses a subset of bits in the read request signal to embed a command (or instruction) for PiM execution; and the second signal is a read response signal. In some implementations, the first signal is a read request signal generated by the memory controller of the SoC; and the second signal is a read response signal generated by the processor in the PiM architecture of the memory device. In some implementations, the first signal is a read request signal generated by the memory controller of the SoC; and the second signal is a read response signal generated by the processor in the PiM architecture of the memory device.

[0015] The computational operation may include a single instruction, and triggering the execution of the computational operation in the memory device includes: triggering the execution of a single instruction stored in the instruction buffer of the PiM architecture in the memory device. The computational operation may include multiple instructions, and triggering the execution of the computational operation in the memory device includes: triggering the synchronous execution of the multiple instructions in the memory device; or triggering the asynchronous execution of the multiple instructions in the memory device.

[0016] Determining the state of the computational operation may include: determining whether an error associated with the computational operation occurred in the memory after the execution of the computational operation was triggered in the memory device. In some implementations, the method further includes: generating a configuration signal at the SoC; and triggering the selection of a specific mode in the memory device based on the configuration signal. The first signal and the configuration signal may be the same signal. The specific mode selected based on the configuration signal may be a mode used by the PiM architecture during the execution of the computational operation in the memory device.

[0017] In some implementations, triggering the selection of a specific mode in the memory device includes: triggering the selection of a first error capture mode, wherein after the execution of the computational operation is triggered in the memory device, the computational operation is paused or stopped in the PiM architecture in response to the detection of an error associated with the computational operation. Triggering the selection of a specific mode in the memory device may also include: triggering the selection of a second, different error capture mode, wherein after the execution of the computational operation is triggered in the memory device, the execution of the computational operation in the memory continues uninterrupted in response to the detection of an error associated with the computational operation.

[0018] In some implementations, the first error capture mode and the second distinct error capture mode each cause the memory device to capture error syndrome data in response to detecting an error associated with the computational operation after triggering the execution of the computational operation in the memory device. The error syndrome data may include information about instructions and data flow operations being executed in the memory at the time the error occurred after triggering the execution of the computational operation in the memory device; and the SoC receives the error syndrome data in response to a request to transfer it from the SoC to the memory device. The memory device performing the computational operation includes a PiM architecture having a processor unit and a mode register coupled to the processor unit.

[0019] Triggering the selection of a specific mode in the memory device may include: triggering the selection of the specific mode by the mode register. In some implementations, receiving the second signal generated by the memory die referencing the memory device includes: receiving the second signal from the memory device using a threshold number of input / output (I / O) pins coupled to the memory. Receiving the second signal generated by the memory die referencing the memory device may include: receiving a success or failure indicator specific to an instruction executed by the processor unit; and receiving an identifier coupled to the mode register of the processor unit executing the instruction. In some implementations, the computational operation triggered to be executed in the memory device is part of computation for a machine learning inference workload.

[0020] One aspect of the subject matter described in this specification can be embodied in a System-on-a-Chip (SoC) including: a processing means; and a non-transitory machine-readable storage means for storing instructions executable by the processing means to cause the execution of operations including: i) generating a first signal at the SoC; ii) triggering the execution of a computational operation in a memory means coupled to the SoC based on the first signal; iii) after triggering the execution of the computational operation in the memory, receiving a second signal generated from the memory means with reference to a memory die of the memory means; and iv) determining the state of the computational operation based on the second signal.

[0021] One aspect of the subject matter described in this specification can be embodied in a computer-implemented method comprising: i) determining a memory access range at a System-on-a-Chip (SoC); ii) generating a write command based on the memory access range; iii) transmitting the write command to the memory device using a control channel coupling the memory device to the SoC; and iv) configuring access control at the memory device based on the memory access range specified by the write command. The access control restricts the PiM computing elements of the memory device to accessing only specific memory portions or memory address locations within the memory device.

[0022] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, configuring access control at the memory device includes configuring row-level memory access constraints at the memory device, wherein the row-level memory access constraints restrict the PiM computing element to accessing a specific range of rows at the memory device.

[0023] Configuring access control at the memory device may include configuring the access control by writing the memory access range into the configuration space of the PiM architecture in the memory device based on the write command. In some implementations, configuring access control at the memory device includes defining a memory portion, memory address, or memory location in the memory device as the only memory portion accessible to the PiM computing element of the memory device.

[0024] The memory access range may include a physical address range, and the method includes: writing the physical address range to a range register in the memory controller of the SoC; and translating the physical address range into row, column, and group addresses of the memory device. In some implementations, generating the write command includes: generating the write command using the memory controller of the SoC. Determining the memory access range may include: determining the memory access range using a trusted agent of the central processing unit (CPU) on the system-on-chip (SoC). The trusted agent of the CPU may include: the CPU core; and the CPU hypervisor.

[0025] One aspect of the subject matter described in this specification can be embodied in a System-on-a-Chip (SoC) including: a processing means; and a non-transitory machine-readable storage means for storing instructions executable by the processing means to cause operations to be performed, including: i) determining a memory access range at the SoC; ii) generating a write command based on the memory access range; iii) transmitting the write command to the memory means using a control channel coupling the memory means to the SoC; and iv) configuring access control at the memory means based on the memory access range specified by the write command. The access control restricts the PiM computing elements of the memory means to access only specific memory portions or memory address locations within the memory means.

[0026] One aspect of the subject matter described in this specification can be embodied in a computer-implemented method comprising: i) determining at a system-on-chip (SoC) refresh requirements for refreshing rows of memory devices coupled to the SoC; ii) generating one or more refresh commands at the SoC based on the refresh requirements; iii) transmitting the one or more refresh commands to a plurality of PiM computing elements of the memory devices coupled to the SoC; and iv) actively (or dynamically) triggering a refresh of memory cells at each of the rows of the memory devices based on the one or more refresh commands transmitted to the plurality of PiM computing elements.

[0027] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, actively triggering a refresh of memory cells at each of these rows of the memory device includes: actively (or dynamically) triggering the refresh of these memory cells at a frequency exceeding the minimum refresh requirement for refreshing a row of the memory device. The plurality of PiM computing elements of the memory device may be used to actively (or dynamically) trigger the refresh of these memory cells at each of these rows at a frequency exceeding the minimum refresh requirement. In some implementations, transmitting the one or more refresh command signals includes: transmitting the one or more refresh command signals within one or more clock cycles generated by the memory controller of the SoC.

[0028] One aspect of the subject matter described in this specification can be embodied in a System-on-Chip (SoC) including: a processing means; and a non-transitory machine-readable storage means for storing instructions executable by the processing means to cause the execution of operations including: i) determining at a system-on-chip (SoC) a refresh requirement for refreshing rows of memory devices coupled to the SoC; ii) generating one or more refresh commands at the SoC based on the refresh requirement; iii) transmitting the one or more refresh commands to a plurality of PiM computing elements of the memory devices coupled to the SoC; and iv) actively (or dynamically) triggering a refresh of memory cells at each of the rows of the memory devices based on the one or more refresh commands transmitted to the plurality of PiM computing elements.

[0029] Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on a computer storage device and configured to perform actions of these methods. A system of one or more computers may be configured such that, in operation, the system performs actions through software, firmware, hardware, or a combination thereof installed on the system. One or more computer programs may be configured to have instructions that, when executed by a data processing device, cause that device to perform actions.

[0030] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0031] Figure 1 This is a block diagram of an example computing system with at least one SoC.

[0032] Figure 2 The example memory in the processor architecture used in the debugging process is shown.

[0033] Figure 3 This illustrates a processor architecture in an example memory used for access control procedures.

[0034] Figure 4 This illustrates an example processor architecture in memory used for the memory refresh process.

[0035] Figure 5 This is an example procedure used to implement the debugging process.

[0036] Figure 6 This is an example procedure used to implement access control procedures.

[0037] Figure 7 This is an example procedure for implementing the memory refresh process.

[0038] Figure 8 An example memory controller and a processor controller in memory are shown that work together to optimize refresh operations of a memory device.

[0039] The same reference symbols and names in different diagrams indicate the same elements. Detailed Implementation

[0040] Figure 1 This is a block diagram of an example computing system 100 including a system-on-a-chip (“SoC 102”). The SoC 102 includes a central processing unit 104 (“CPU 104”), a memory controller 105, shared memory 106 (“memory 106”), a resource manager 108, and IP / circuit blocks 110. In some implementations, system 100 may include multiple SoCs, and any description of SoC 102 will equally apply to each of the multiple SoCs that may be included in system 100.

[0041] CPU 104 may be a general-purpose CPU (e.g., a single-core or multi-core CPU). CPU 104 generates one or more indicators, such as an app launch indicator or function call triggered in response to the execution or launch of an application on a user device. For example, the application may be a camera application that uses an imaging sensor to generate image data or a game application that requires significant memory and graphics processing resources to render the game's graphical content. CPU 104 also generates one or more application values, such as pixel values ​​or frame rates. Application values ​​may be associated with function calls, may describe events that occur during the execution of the application, or both.

[0042] Memory 106 is system memory, shared memory, or both. Figure 1In the example, memory 106 is depicted as being outside circuit block 110. However, memory 106 may include memory portions that are i) specific to circuit block 110, ii) outside circuit block 110, or iii) both. Memory 106 may be random access memory of SoC 102, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0043] In some implementations, aspects of memory 106 are configured as shared scratchpad memory, which supports parallel access to its memory resources by two or more processors of circuitry 110. Memory 106 may also include various other types of memory, such as high-bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0044] Resource Manager 108 is implemented in both hardware and software. Aspects of Resource Manager 108 may also be implemented as firmware of SoC 102 or firmware of a device of SoC 102 (such as a DRAM memory device or CPU 104). Resource Manager 108 is a processor-in-memory (PiM) resource manager (“PiM Resource Manager 108”) that includes control logic implemented in hardware, software, or both. For example, PiM Resource Manager 108 may include resources implemented in hardware, such as flip-flops, registers, buffers, etc.; and control logic implemented in software (e.g., programmed code).

[0045] Circuit block 110 typically includes various IP devices, such as processors, processor cores, or dedicated processing devices. For example, circuit block 110 may include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. Circuit block 110 is alternatively referred to as IP block 110, where an IP block may include one or more proprietary hardware elements. For example, each of ISP 112, TPU 114, DSP 116, and GPU 118 may be a corresponding proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0046] One or more aspects of the PiM resource manager 108 may be implemented as software routines (or modules) of the CPU 104 that utilize one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 may be configured as an instruction and vector data processing engine that processes data obtained from the system memory of the SoC 102, such as memory 106. In some implementations, each processor of the SoC 102 (e.g., ISP 112, DSP 116, TPU 114, GPU 118) includes multiple cores, and the CPU 104 and / or the PiM resource manager 108 may generate control signaling to manage and distribute memory-intensive computational operations on memory devices 122 (e.g., DRAM) to minimize the processing load at each core of the processor. Control signaling is routed at system 100 using an example bus 120 of the SoC 102. Control signaling may include commands, requests, data, instructions, or combinations thereof.

[0047] PiM Resource Manager 108 collaborates with CPU 104, memory controller 105, and memory controller 107 to dynamically control and manage one or more in-memory compute (CIM) operations. In some implementations, CIM operations are performed at SoC 102 to support heterogeneous compute operations across two or more processing units, including IP block 110, CPU 104, or both. More specifically, PiM Resource Manager 108 is configured to generate control signaling and use one or more discrete signal values ​​of this control signaling to manage and execute debugging operations, access control operations, and memory refresh operations at memory device 122.

[0048] System 100 includes an example memory device 122. Memory device 122 may include a plurality of memory dies. For example, memory device 122 may include... N N is a memory die, where N is an integer greater than 1. The memory device 122 may be a dynamic random access memory (DRAM) or double data rate (DDR) synchronous DRAM (SDRAM). The memory device 122 is configured to perform or support various types of PiM operations, CiM operations, and near-memory computation operations (“MnC operations”). The memory device 122 uses its multiple PiM computation elements to perform or support these operations, which are referred to below. Figures 2 to 4 To describe.

[0049] SoC 102 collaborates with memory device 122 to perform computations across one or more memory dies of memory device 122. These computations can be used for operations or workloads involving one or more processors at IP block 110. Additionally, these computations can be for heterogeneous operations across multiple processors, multiple IP blocks 110, or both. In at least one example, memory device 122 may be external to SoC 102, while in another example, memory device 122 may be internal to SoC 102.

[0050] exist Figure 1 In the example, system 100 and SoC 102 are integrated circuits for example user / client device 130, consumer electronic device, or mobile device, each of which may include items such as smartphone 130a, tablet computer 130b, laptop computer 130c, smartwatch, or wearable device 130d. Device 130 may also include other items such as e-notebook computer, netbook computer, smart speaker, or mobile computer. In some implementations, system 100 and SoC 102 are integrated circuits for desktop computers, web servers, or associated cloud-based assets.

[0051] Figure 2 An example in-memory processor architecture 200 is shown for a debugging process performed using SoC 102, memory device 122, or both. Figure 2 In one example, memory device 122 includes: a first memory die-1, which includes one or more memory arrays 210-1; and a second memory die-2, which includes one or more memory arrays 210-2.

[0052] PiM architecture 200 includes multiple computing elements. For example, PiM architecture 200 includes mode registers 204-1 and 204-2 located at memory die-1 and memory die-2, respectively. PiM architecture 200 further includes processing units 206-1 and 206-2 located at memory die-1 and memory die-2, respectively. Each processing unit 206-1 and 206-2 may include a processor, processor unit, or processor core, such as a CPU. Each processing unit 206-1 and 206-2 may also include example computing units, such as an arithmetic logic unit (ALU) or a multiply-accumulate unit (MAC). PiM architecture 200 and other PiM architectures disclosed herein may include multiple control and status registers (CSRs), which represent auxiliary registers used for reading status and changing the configuration and mode of operations in the PiM architecture.

[0053] In some implementations, the PiM architecture 200 is included in the memory device 122 as a plurality of discrete integrated circuits, wherein each integrated circuit is local to a given memory die (e.g., die-1 and die-2) and interacts or communicates with an array of memory cells at that memory die. For example, the PiM architecture 200 may include computing elements replicated and distributed across each of the memory dies in the memory device 122. In some other implementations, the PiM architecture 200 is included in the memory device 122 as a single integrated circuit that interacts or communicates with each memory die of the memory device 122 (including the array of memory cells at each memory die).

[0054] PiM operations can include standard CPU functions, while CiM and MnC operations can include standard arithmetic operations, such as computations typically performed by the ALU or MAC. CiM and MnC operations can also include the computational capabilities of the TPU 114, such as multiplication and addition operations for matrix mathematics, vector computation, linear algebra, and dot-matrix accumulation. In some implementations, each of the PiM, CiM, and MnC operations is performed to support machine learning computations, neural network computations, or both.

[0055] The PiM architecture 200 may include registers or other memory portions for storing error data 220-1, 220-2 for corresponding memory dies. Each of the error data 220-1, 220-2 describes an error corrector for an error that occurred at the corresponding memory die of the memory device 122 during computation operations. Mode registers 204-1, 204-2 are used to control or trigger the selection of specific modes in the PiM architecture, such as error capture mode, continuous / uninterrupted execution mode, etc.

[0056] In some implementations, a specific mode is selected based on the bit values ​​of mode registers 204-1 and 204-2. For example, to trigger or select one or more error capture modes, a single bit or a sequence of bits is defined for use in the mode registers. The bit values ​​of mode registers 204-1 and 204-2 are set via commands / instructions received from the SoC 102, local commands / instructions executed by the processing unit of the PiM architecture 200, or both. For example, PiM mode register values ​​can be programmed using a mode register write command (MRW), while other PiM registers (such as the PiM CSR) can be programmed using a PiM control register write command.

[0057] A subset of bits in the example read request signal can be used to trigger an error capture mode of the PiM architecture 200. In some implementations, a specific error capture mode is triggered and / or configured in the PiM architecture 200 when a command from the SoC 102 triggers the bulk execution of multiple PiM instructions in the memory device 122. In this (and other) implementations, a first error capture mode of the PiM architecture is configured to suspend the execution of the PiM operation in response to the detection or occurrence of an error, while a second, different error capture mode of the PiM architecture is configured to continue the execution of the PiM operation in response to the detection or occurrence of an error. See below for reference. Figure 5 The example procedure describes additional aspects of the error capture mode in more detail.

[0058] Figure 3 An example in-memory processor architecture 300 is shown for an access control procedure executed using SoC 102, memory device 122, or both. PiM architecture 300 may be an extension of PiM architecture 200. Figure 3 In the example, the memory controller 105 of SoC 102 includes a range register 304, and the memory device 122 includes a first row limiter 310-1 and a second row limiter 310-2. The range register 304 is configured to store address values ​​corresponding to physical memory locations of the memory device 122. Each of the first row limiter 310-1 and the second row limiter 310-2 may be implemented in hardware, software, or both. In some implementations, the first row limiter 310-1 and the second row limiter 310-2 define the configuration space of the PiM architecture.

[0059] SoC 102 determines the memory access range, and memory controller 105 generates a command that transmits the memory access range to memory device 122. The memory access range may be represented as a physical address range. SoC 102 uses a trusted agent of CPU 104 to determine the memory access range. The trusted agent may be a core of CPU 104, a hypervisor of CPU 104, or both. The memory access range is represented or expressed as a physical address range by CPU 104, and SoC 102 writes the physical address range to range register 304 in memory controller 105.

[0060] Memory controller 105 uses range register 304 to maintain a range of physical address values ​​of memory locations accessible to computing elements of the PiM architecture in memory device 122. Memory controller 105 uses the physical address range stored in range register 304 to generate and transmit commands, such as access control messages, to compute PiM architecture elements in memory device 122. The command may be a write command for configuring memory access control at memory device 122 based on the memory access range specified by the command. In some implementations, memory device 122 is a DRAM device including one or more configuration mode registers. In this implementation, the write command may be an MR write command corresponding to an MR transaction for configuring a particular access control mode. For example, the access control mode is configured based on bit values ​​set in the configuration mode register of memory device 122.

[0061] A write command defines a range of physical addresses accessible by the PiM system of the memory die in memory device 122. The PiM system includes or is based on a corresponding PiM architecture 300. In some implementations, memory controller 105 is configured to translate this physical address range into row, column, and group addresses of memory device 122, and then generate a write command to transfer the row, column, and group addresses to memory device 122. A row limiter can use the row, column, and group addresses to generate control signals that define or "cut out" a portion of memory as a unique portion of memory accessible to the PiM computing elements of the memory device. This portion of memory may be a set of rows in a corresponding DRAM group of memory device 122.

[0062] Each row limiter 310-1, 310-2 can interact with corresponding processing units 206-1, 206-2 to configure memory access control at memory device 122 based on the row, column, and group addresses indicated by the write command. For example, row limiter 310-1 can generate a row limit signal based on the memory access range specified by a command executed at processing unit 206-1 of the corresponding memory die-1. Similarly, row limiter 310-2 can generate a row limit signal based on the memory access range specified by a command executed at processing unit 206-2 of the corresponding memory die-2.

[0063] In some implementations, computational elements such as row limiters 310-1, 310-2 and corresponding processing units 206-1, 206-2 of the PiM architecture are configured for row-level constraints. For example, the PiM system can be configured to access any column and / or any group within memory device 122, but is limited to accessing only a subset of rows within a group. Computational elements of the PiM architecture 300 can be configured to implement different types of access constraints using various computational elements of the PiM architecture 300. In some implementations, memory access control is configured by writing the memory access range into a mode register of the configuration space in memory device 122 based on write commands.

[0064] Figure 4 An example in-memory processor architecture 400 is shown for a memory refresh process performed using SoC 102, memory device 122, or both. PiM architecture 400 may be an extension of PiM architecture 200 and PiM architecture 300. Figure 4 In the example, SoC 102 communicates with computing elements of PiM architecture 400 to optimize the refresh process performed at memory device 122. For example, computing elements such as mode registers 204-1, 204-2 and corresponding processing units 206-1, 206-2 are used to actively (or dynamically) trigger refreshes of memory cells at each of the rows across each DRAM bank of memory die-1.

[0065] In some implementations, the refresh process is optimized by actively (or dynamically) triggering the refresh of those memory cells at a frequency exceeding the minimum refresh requirement for refreshing rows of memory device 122. For example, the internal controller of processing unit 204-1 can actively or dynamically perform refreshes based on the frequency or refresh rate of a clock cycle generated by memory controller 105. For instance, the internal controller of the PiM architecture can actively perform refreshes at 16-millisecond (ms) intervals instead of the minimum 32-ms interval that the memory device may require. The refresh operation is represented by a refresh signal triggered by a refresh command processed at the PiM architecture of memory device 122. See below for reference. Figure 8 The example provides a more detailed description of the refresh process and refresh optimization.

[0066] For clarity, the features, devices, and / or computing elements described with reference to PiM architectures 200, 300, and 400 may correspond to or be integrated into a single PiM architecture. Furthermore, each of PiM architectures 200, 300, and 400 may include more or fewer process units, mode registers, and / or row limiters, regardless of the number of these elements shown in the corresponding diagrams including these elements.

[0067] Figure 5 This is an example process 500 for performing a debugging process. Process 500 is implemented or executed at system 100 using at least the SoC 102 and memory device 122 described above. Therefore, the description of process 500 will refer to the computing resources of system 100 described above. In some examples, the steps or actions of process 500 are implemented by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and may be executed by one or more of the processors described in this specification or other resources.

[0068] Referring again to process 500, SoC 102 generates a first signal (502) at the SoC. For example, the first signal may be a read request signal generated by memory controller 105. In some implementations, in addition to the primary purpose of requesting, reading, or obtaining data from memory device 122, the read request signal (and its associated bit data structure) may also be reused for the secondary purpose of providing a trigger for the execution of computational operations in the PiM module of memory device 122.

[0069] SoC 102 triggers the execution of a computational operation in a memory device coupled to SoC 102 based on a first signal (504). For example, the computational operation may be a CiM operation, in which neural network input stored at memory device 122 is retrieved and used as an operand for a multiplication or addition operation performed at an ALU, MAC, or compute unit of a PiM architecture 200, 300, 400 in memory device 122.

[0070] In some implementations, the first signal is an execution signal defined based on an execution protocol managed and / or controlled by CPU 104, PiM resource manager 108, or both. In relevant implementations, the read (or read request) signal is defined based on a bit structure, such as 16-bit, 32-bit, etc. SoC 102 can be configured to "piggyback" example execution commands onto existing read (or write) signals of the PiM architecture 200, 300, 400 transmitted to memory device 122. More specifically, an example 32-bit read (or write) request signal may include one or more unused bits (e.g., one or two bits), and SoC 102 may use one or more unused bits of this signal to "piggyback" execution commands onto the read / write signal.

[0071] For example, SoC 102 may: i) define a high-order bit value (binary "1") to indicate the initial position in the unused bit sequence of the 32-bit read / write signal; ii) execute a command embedded in the read / write signal by setting the high-order bit value at that initial / specific position in the unused bit sequence; and iii) use the high-order bit value as a trigger / command for performing computational operations at memory device 122. SoC 102, the host device (e.g., TPU 114), or the memory controller 105 may also embed additional execution or configuration commands into the read / write signal by setting the bit values ​​of other bits in the sequence to specific high-order or low-order bit values ​​(binary "0").

[0072] In some cases, the read request signal specifies the actual read address, which specifies the memory location from which data is to be obtained or read, along with additional operation trigger bits, while the read response signal also returns the read or requested data, and error / status information indicating the execution status of the PiM command embedded in the previous read request signal. The execution command may be embedded in the read / write signals transmitted directly from SoC 102 to the PiM element in memory device 122. In some other cases, the execution command may be embedded in the read / write signals transmitted to non-PiM elements of memory device 122. Embedding the execution command in signals transmitted to non-PiM elements is useful for minimizing wasted cycles and interrupts at the PiM architecture of memory device 122, which conserves and / or maximizes clock cycle usage in system 100.

[0073] To further enhance efficiency, including cycle usage across system 100, some (or all) read / write signals and individual execution commands can be configured to trigger the execution of multiple instructions, operations, and / or computational operations at the PiM architecture of memory device 122. For example, commands or instructions from SoC 102 can be passed to memory device 122 to trigger the instruction register of PiM architecture 200 within memory device 122. N The execution of one (e.g., multiple) instructions. In this example and others, N Can be an integer greater than 1. Commands / instructions sent from SoC 122 can be configured, for example, by... N The value is embedded or defined in the command / instruction to define the number of instructions.

[0074] In some implementations, the PiM resource manager 108 is configured to determine a threshold number of commands required to ensure that a subset of the computations needed for the PiM architecture to continuously perform a given task. This continuous execution of computation may correspond to or be achieved through the synchronous execution of instructions described below. The PiM resource manager 108 may determine the threshold number of commands based on the execution bandwidth of a particular PiM architecture at memory device 122. In some implementations, the PiM execution bandwidth is defined by the minimum number of commands / instructions necessary to ensure that the PiM architecture can continuously perform the computations required for a given workload.

[0075] SoC 102 is configured to provide a subset of instructions buffered at the example instruction buffer of the PiM architecture. In addition to the execution protocol described above, SoC 102 may also include an instruction protocol managed and / or controlled using CPU 104, PiM resource manager 108, or both. The instruction protocol may define instruction signals for delivering a subset of instructions buffered at the instruction buffer of the PiM architecture in memory device 122. For instruction tracing, the mode register of PiM architecture 200 may be configured as a program counter to track instruction execution, delays, and / or pauses.

[0076] In some implementations, SoC 102 is configured to construct a subset of commands that includes opcodes for encoding a specific type of ML / NN computation that the PiM module will perform on a particular data sequence. SoC 102 may also embed address configuration commands in read request signals, or simply define new or unique 16-bit, 32-bit, or 64-bit communication / command signals for configuring PiM / CiM operations in the PiM architecture 200. For example, a new 32-bit command signal may indicate the range of memory addresses storing data (e.g., operands) for a specific PiM / CiM operation.

[0077] Example computational operations performed at the PiM architecture 200 may include or span multiple instructions. For a subset of instructions buffered in the PiM module of the PiM architecture 200, the SoC 102 and / or the PiM module may trigger synchronous execution of the instructions to perform the computational operation. Alternatively, for a subset of instructions buffered in the PiM module of the PiM architecture 200, the SoC 102 and / or the PiM module may also trigger asynchronous execution of the instructions to perform a given computational operation. In some implementations, aspects of the instruction protocol and execution protocol are integrated into a single protocol. For example, the SoC 102 may be configured to integrate or combine features used to define processing instructions and execution commands into a single protocol.

[0078] SoC 102 receives a second signal (506) from the memory. The second signal may be a read response signal generated at or with reference to the memory die of memory device 122 after triggering the execution of a computation operation in the memory device. In some implementations, the second signal (e.g., a read response signal) conveys or includes data describing one or more errors that occurred during the computation operation. SoC 102 determines the state of the computation operation (508) based on the second signal. System 100 may use the second signal generated at or with reference to memory device 122 to convey success / failure information and certain status information.

[0079] For example, receiving a second signal (e.g., a read response or other type of signal) includes receiving an indicator indicating whether a specific instruction executed by processing units 206-1, 206-2 has succeeded or failed. This indicator is specific to the instruction executed by the processing unit and includes information about the execution status of that instruction. In some implementations, the second signal is a unique response signal, distinct from the read response signal, generated within the PiM architecture 200 using processing units 206-1, 206-2. In some cases, a subset of bits can be generated using means or computing elements of the PiM architecture 200, wherein the bit subset may be included in or appended to a data structure corresponding to the second / response signal. The bit subset may encode parameters indicating whether a specific instruction was executed successfully, failed, or stalled. These parameters may also indicate that the execution of the specific instruction is delayed.

[0080] In some implementations, receiving the second signal includes receiving an identifier for a specific mode register (“Mode Register ID”) coupled to processing units 206-1, 206-2 for executing instructions. Mode registers 204-1, 204-2 may specify to a specific processing unit or PiM control logic the number of computational instructions assigned to processing units 206-1, 206-2 for execution. In other words, the mode registers of the PiM architecture 200 are configured to inform the PiM control logic how many instructions its processing units will execute for a given workload. Additionally, the mode registers may include error mode configuration values ​​specifying the error capture mode for a given workload. In some implementations, mode registers 204-1, 204-2 are configured to capture or track the position of the program counter within the list of instructions executed within the PiM architecture 200.

[0081] Read response signals (e.g., a second signal) can be transmitted using one or more data lines (e.g., D / Q lines). System 100 can be configured to reuse data lines typically reserved for providing or retrieving data stored in memory device 122. In some implementations, system 100 uses, for example, mode registers 204-1, 204-2 to include or select a specific PiM mode, wherein a specific number of data lines are used (or will be used) to indicate the success or failure of a given operation or instruction. For example, in this specific PiM mode, system 100 can switch the specific number of data lines required to indicate the success or failure of an operation or instruction.

[0082] The data lines represent the communication interface between SoC 102 and memory device 122, where the interface comprises 16 lines. Typically, data lines are alternatively referred to as D / Q lines or I / O lines. In some implementations, SoC 102 is configured to receive response signals (e.g., a second signal) from memory device 122 using a threshold number of input / output (IO) pins coupled to memory device 122. For example, to indicate whether a PiM instruction was successfully executed, system 100 may be configured to toggle only one of the D / Q lines instead of toggling all 16 lines. For example, when indicating success or failure of instruction execution at memory device 122, system 100 may toggle a single (or fewer than all) D / Q line as a power gating or power control mechanism for the system. In some implementations, system 100 may use short / shorter data bursts to transmit certain status information.

[0083] Errors occurring during computation operations can be captured based on the error capture mode of the PiM architecture in memory device 122. A specific error capture mode defines how the PiM architecture captures errors occurring during the execution of memory-constrained computation operations as error data 220-1, 220-2. A specific mode can be selected based on configuration signals passed from SoC 102 to the PiM architecture. For example, resources of SoC 102 can generate configuration signals at the SoC and transmit these configuration signals to the PiM architecture of memory device 122.

[0084] In some implementations, the configuration signal is passed to the PiM architecture as a read / write request signal or some other signal. For example, the configuration signal can be embedded in an existing signal (e.g., a read / write signal) using one or more unused bits of that existing signal. Instead of embedding configuration information into an existing read / write signal, SoC 102 can also be configured to include a separate / unique configuration signal to transmit computed configuration settings to PiM architecture 200. SoC 102 is configured to trigger the selection of a specific mode in memory device 122 based on the corresponding values ​​of one or more bits representing the configuration signal.

[0085] The example error capture mode can be configured such that after execution of a computational operation is triggered at the PiM architecture of memory device 122, the computational operation is automatically paused or stopped in memory device 122 in response to the detection of an error associated with the computational operation. For example, the computational operation can be paused or stopped using processor units 206-1, 206-2, or the relevant control logic of the PiM architecture. In another example, a second, different error capture mode can be configured such that after execution of a computational operation is triggered at memory device 122, the execution of the computational operation occurs uninterrupted in response to the detection of an error associated with the computation.

[0086] During this continuous error capture mode, the error capture and detection logic of the PiM architecture can be configured with efficiency biases, for example, to conserve memory resources and capture PiM architecture cycles. The efficiency biases can be configured such that after detecting a first instance of a given error type, the PiM architecture discards capturing a second instance of the same (or substantially similar) error type. In some implementations, the error capture and detection logic of the PiM architecture includes an overflow flag associated with a given error type. The overflow flag is used to indicate that two or more instances of this given error type have been detected or have occurred.

[0087] For each error capture mode, the PiM architecture of memory device 122 is configured to capture error correction sub-data during the execution of a computational operation and in response to the detection of an error associated with that computational operation. The error correction sub-data includes information about instructions and / or data flow operations that have been executed or are being executed when the error occurs during the computational operation. In some implementations, the error correction sub-data is generated based on a stack or memory dump operation performed to recover instruction and / or data flow information for subsequent debugging operations.

[0088] Error correction subdata can be configured to capture one or more of the following: the index or address of the instruction that caused the specific error, the cause of the error (e.g., undefined instruction, out-of-bounds access, etc.), and the memory address that the instruction attempted to access. Error correction subdata (e.g., error data 220-1, 220-2) can be captured and / or stored in the example error register of the PiM architecture, the mode registers 204-1, 204 of the PiM architecture, or both.

[0089] In some implementations, mode registers 204-1, 204 may be configured as overflow resources of the error registers of PiM architectures 200, 300, 400 to capture error correction sub-data exceeding the storage capacity of the error registers. SoC 102 is configured to retrieve error correction sub-data from the error registers, for example, using read / write response signals as described above. For example, error correction sub-data may be sent from or transferred to SoC 102 upon request from SoC 102. In some implementations, error correction sub-data, or portions thereof, may be embedded in a response signal using one or more unused bits of the response signal.

[0090] Figure 6 This is an example process 600 for performing access control procedures. Process 600 is implemented or executed at system 100 using at least the aforementioned SoC 102 and memory device 122. Therefore, the description of process 600 will refer to the aforementioned computing resources of system 100. In some examples, the steps or actions of process 600 are implemented by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and may be executed by one or more of the processors described in this specification or other resources.

[0091] Referring again to process 600, SoC 102 determines the memory access range (602). For example, CPU 104 or PiM resource manager 108 may dynamically determine the memory access range based on the type of PiM operation or CiM operation being performed at memory device 122. In some implementations, CPU 104 or PiM resource manager 108 may dynamically determine refresh requirements based on the type of memory device 122 coupled to SoC 102.

[0092] SoC 102 generates a write command (604) based on a memory access range. For example, SoC 102 may use CPU 104 and / or PiM Resource Manager 108 to generate control signals that cause memory controller 105 to generate a write command. In some implementations, memory controller 105 is configured to read a memory access range (e.g., an address range) specified in its range register 304 and then generate a write command to pass or transfer that address range to row limiters 310-1, 310-2 in PiM architecture 300.

[0093] SoC 102 uses the control channel of the coupled memory device 122 to transmit write commands to memory device 122 (606). In some implementations, memory controller 105 interprets or translates an address range into one or more rows, groups, or columns of memory device 122, and uses the corresponding rows, groups, or columns for the address range to pass the address range to row limiters 310-1, 310-2.

[0094] System 100 configures access control at memory device 122 based on write commands (608). More specifically, SoC 102 interacts with memory device 122 to configure access control at memory device 122 based on the memory access range specified by the write command. In some implementations, the PiM architecture includes a configuration space containing row limiters 310-1, 310-2 and one or more configuration mode registers. System 100 can configure access control at memory device 122 by writing a memory access range into the configuration space of the PiM architecture based on a write command. For example, a bit sequence of a write command (e.g., an MR write command) can configure a specific access control mode by setting certain values ​​in the configuration mode register.

[0095] In some implementations, SoC 102 configures access control at memory device 122 by configuring row-level memory access constraints at specific memory dies of memory device 122. Row-level memory access constraints restrict PiM computing elements to accessing a specific range of rows at a given group of memory dies. SoC 102 configures row-level memory access constraints in coordination with the PiM architecture, for example, by triggering an access control configuration using bit, binary, or signal values ​​of write commands.

[0096] Access control restricts the PiM computing elements of memory device 122 to access only memory cells or specific portions of the memory array (610) within the memory device. For example, a bit sequence of a write command can transmit a specific address range used by row limiters 310-1, 310-2 to constrain (or restrict) access to specific rows, columns, or groups of memory device 122, as indicated by the write command.

[0097] Figure 7 This is an example process 700 for performing a memory refresh procedure. Like other processes described herein, process 700 is implemented or executed at system 100 using at least the aforementioned SoC 102 and memory device 122. Therefore, the description of process 700 will refer to the aforementioned computing resources of system 100. In some examples, the steps or actions of process 700 are implemented by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and may be executed by one or more of the processors described in this specification or by other resources.

[0098] Referring again to process 700, SoC 102 determines refresh requirements (702) for refreshing rows of memory devices coupled to the SoC. For example, CPU 104 or PiM Resource Manager 108 may dynamically determine refresh requirements based on the type of memory device 122 coupled to SoC 102, such as whether memory device 122 is a specific type of DRAM or DDR SRAM. CPU 104 or PiM Resource Manager 108 may also dynamically determine refresh requirements based on the attributes of memory cells within memory device 122. In some implementations, refresh requirements are predetermined and stored locally at SoC 102 (e.g., in firmware).

[0099] SoC 102 generates one or more refresh commands based on refresh requirements (704). SoC 102 transmits these refresh commands to multiple in-memory processing (PiM) computing elements coupled to the memory device of the SoC (706). For example, SoC 102 transmits these refresh commands to at least memory arrays 210-1 and 220-2 of memory device 122. SoC 102 actively triggers a refresh of a memory cell at one or more rows of the memory device (708). As indicated above, a refresh is an example signaling operation performed on certain memory cells or rows of memory device 122 to maintain the integrity of data stored in memory cells at each row of the memory device.

[0100] System 100 proactively optimizes the refresh process, for example, by triggering refreshes of those memory cells at a frequency exceeding the minimum refresh requirement for refreshing rows of memory device 122. In some implementations, system 100 is configured to optimize by tracking memory access patterns of PiM commands or operations executed at the PiM architecture: i) the timing of issuing and / or executing refresh commands, and ii) the total number of refresh commands. For example, system 100 can determine refresh schedules that meet or exceed the minimum refresh requirements of memory device 122 by analyzing past, ongoing, or upcoming PiM operations and their corresponding memory access patterns.

[0101] The SoC 102 triggers refreshes based on one or more refresh commands transmitted to the compute elements of the PiM architecture. For example, compute elements such as mode registers 204-1, 204-2 and corresponding processing units 206-1, 206-2 can be used to actively trigger refreshes of memory cells at each of the rows across each DRAM bank of the memory die-1.

[0102] In some implementations, for a given PiM operation, the refresh logic of SoC 102 can be configured to include a refresh signal in the PiM execution command (e.g., a read request signal). The refresh signal can be passed to memory device 122 such that the corresponding operand arrays (odd / even) accessed during the PiM operation receive the refresh signal after the execution of the PiM operation involving those operand arrays. For example, a refresh operation can be performed on at least two rows of the memory die after all computational operations have been performed for a given task or workload. In other implementations, PiM commands initiated in the PiM architecture and executed locally, for example via processing units 206-1, 206-2, can be configured to embed a refresh command to trigger a refresh of the operand arrays.

[0103] The corresponding steps of process 500, process 600, or process 700 can be executed at a hardware integrated circuit as part of a larger computational operation to generate machine learning (ML) output, including the output of a neural network layer that implements one or more ML models. For example, this output can be part of the computation for an ML task or inference workload used to generate image processing, speech processing, or image recognition output.

[0104] As described above, a portion of the integrated circuit may include a dedicated neural network processor or a hardware ML accelerator configured to accelerate computation to generate different types of data processing outputs. In some implementations, one or more of PiM operations, CiM operations, or MnC operations are executed by the memory device 122 to support or implement accelerated computation to generate different types of data processing outputs.

[0105] Figure 8 An example refresh system 800 is shown, comprising a memory controller 105, a host processor 802, and a PiM architecture 400. The refresh system 800 is a subsystem of system 100. Figure 8 In the example, the host processor 802 may be an application-specific integrated circuit, such as TPU 114, and includes an example PiM controller 814 that cooperates with memory controller 105 to optimize refresh operations of the memory device. In some implementations, TPU 114 is configured to execute AI / ML models, such as large language models (LLMs). TPU 114 uses PiM controller 814 and memory controller 105 to offload a portion of the computation used to execute LLM workloads to PiM architecture 400.

[0106] PiM controller 814 generates configuration information, computation commands, and PiM execution instructions 806 to offload computational tasks to PiM architecture 400. PiM controller 814 collaborates with memory controller 105 to transfer the configuration information, computation commands, and PiM execution instructions 806 (collectively, “PiM execution instructions 806”) to PiM architecture 400. PiM execution instructions 806 may include information related to PiM internal data paths and control, such as operation type, operands, and execution mode. Memory controller 105 uses this information generated by host processor 802 and PiM controller 814 to infer a subset of future memory banks that host processor 802 may access to execute part of its ML workload at PiM architecture 400.

[0107] For example, memory controller 105 includes refresh optimization logic 804 that configures and executes the data refresh process to optimize when and how refresh signals 810 are issued across memory banks 830 in the memory device. Refresh optimization logic 804 is configured to schedule refresh intervals for memory banks based on operation type and calculates the duration indicated by PiM execution instructions 806. Specifically, refresh optimization logic 804 may determine an optimal refresh interval 820 for different workloads that maximizes the total PiM computation time (e.g., by minimizing PiM computation downtime), resulting in an overall performance enhancement at PiM architecture 400.

[0108] The memory controller 105 utilizes its timing and scheduling module 808 to optimize the timing, sequencing, and / or scheduling of refresh operations performed at memory bank 830. For example, refresh optimization logic 804 uses constraints enforced by its timing and scheduling module 808 as criteria to optimize its refresh control behavior. In some implementations, the memory controller 105 transmits a refresh control signal 810 that can establish a refresh configuration (e.g., refresh_config) at PiM architecture 400. The refresh control signal 810 can be generated based on the timing and scheduling module 808 to optimize the timing, sequencing, and / or scheduling of refresh operations performed at memory bank 830.

[0109] For example, memory controller 105 optimizes refresh timing and scheduling to minimize interruptions to ML inference computations at PiM modules that process data obtained from memory banks undergoing periodic refreshes. PiM controller 814 collaborates with refresh optimization logic 804 to evaluate the performance and power impact of different refresh rates and determine the refresh scheme that minimizes the performance and power overhead of a given refresh operation. In some implementations, each of PiM controller 814 and refresh optimization logic 804 is configured to optimize refresh execution using known time windows. For example, PiM controller 814 may identify time windows that coincide with natural pauses (e.g., group off / on) in ML inference computations to optimize the execution of group-by-group refresh operations.

[0110] The embodiments of the subject matter and functional operation described in this specification may be implemented using digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or for controlling the operation of a data processing device.

[0111] Alternatively or additionally, these program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by a data processing device. Computer storage media may be machine-readable storage devices, machine-readable storage substrates, random or serial access memory devices, or combinations thereof.

[0112] The term "computing system" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. Devices may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, devices may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0113] Computer programs (which may also be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including as stand-alone programs or modules, components, subroutines, or other units suitable for use in a computing environment.

[0114] Computer programs may, but do not necessarily, correspond to files in a file system. A program may be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinating files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers located in one location or distributed across multiple locations and interconnected via a communication network.

[0115] The processes and logic described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic can also be executed by dedicated logic circuitry, and the device can also be implemented as dedicated logic circuitry, such as FPGA (Field-Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), or GPGPU (General-Purpose Graphics Processing Unit).

[0116] Computers suitable for executing computer programs include, for example, central processing units (CPUs) that may be based on general-purpose or special-purpose microprocessors or both, or any other type. Typically, the CPU receives instructions and data from read-only memory or random access memory or both. Some components of a computer are the CPU for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0117] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0118] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., an LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's client device.

[0119] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server; or middleware components, such as an application server; or front-end components, such as a client computer with a graphical user interface or a web browser, through which a user can interact with the implementation of the subject matter described in this specification; or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or media. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0120] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established through computer programs running on the respective computers that have a client-server relationship with each other.

[0121] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features may be removed from the claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0122] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or to perform all of the shown operations, in order to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0123] Specific embodiments of this subject matter have been described. Other embodiments are also within the scope of the appended claims. For example, the actions listed in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, comprising: The first signal is generated at the system-on-chip (SoC); Based on the first signal, the execution of a computational operation is triggered in the memory device coupled to the SoC; After the execution of the computation operation is triggered in the memory device, a second signal generated by referring to the memory die of the memory device is received from the memory device; as well as The state of the computation operation is determined based on the second signal.

2. The method of claim 1, wherein: The first signal is a read request signal, which uses a subset of bits in the read request signal to embed a command in the processor in the memory ("PiM"); and The second signal is the read response signal.

3. The method as described in claim 1 or 2, wherein: The first signal is a read request signal generated by the memory controller of the SoC; and The second signal is a read response signal generated by the processor in the PiM architecture of the memory device.

4. The method as described in any one of claims 1 to 3, wherein, The computational operation comprises a single instruction, and triggering the execution of the computational operation in the memory device includes: The execution of a single instruction in the instruction buffer of the PiM architecture stored in the memory device is triggered.

5. The method as described in any one of claims 1 to 3, wherein, The computational operation includes multiple instructions, and triggering the execution of the computational operation in the memory device includes: The synchronous execution of the plurality of instructions is triggered in the memory device; or The asynchronous execution of the plurality of instructions is triggered in the memory device.

6. The method as described in any of the preceding claims, wherein, Determining the state of the computation operation includes: Determine whether an error associated with the computational operation occurs in the memory after the execution of the computational operation is triggered in the memory device.

7. The method of any of the preceding claims, further comprising: A configuration signal is generated at the SoC; as well as The selection of a specific mode is triggered in the memory device based on the configuration signal.

8. The method of claim 7, wherein, The first signal and the configuration signal are the same signal.

9. The method of claim 7, wherein, The specific mode selected based on the configuration signal is the mode used by the PiM architecture during the computational operation performed in the memory device.

10. The method of claim 7, wherein, Triggering the selection of a specific mode in the memory device includes: The selection of a first error capture mode is triggered, wherein, after the execution of the computation operation is triggered in the memory device, the computation operation is paused or stopped in the memory in response to the detection of an error associated with the computation operation.

11. The method of claim 10, wherein, Triggering the selection of a specific mode in the memory device includes: The selection of a second, different error capture mode is triggered, wherein, after the execution of the computational operation is triggered in the memory device, the execution of the computational operation in the memory continues uninterrupted in response to the detection of an error associated with the computational operation.

12. The method of claim 11, wherein, The first error capture mode and the second different error capture mode each cause the memory device to capture error correction sub-data in response to detecting an error associated with the computational operation after the execution of the computational operation is triggered in the memory device.

13. The method of claim 12, wherein: The error correction sub-data includes information about the instructions and data stream operations being executed when the error occurs in the memory after the execution of the computational operation is triggered in the memory device; and In response to a request to transfer data from the SoC to the memory device, the error correction sub-data is received by the SoC.

14. The method of claim 12 or 13, wherein, The memory device that performs the computational operation includes: A processor architecture in memory, the processor architecture in memory having a processor unit and a mode register coupled to the processor unit.

15. The method of claim 14, wherein, Triggering the selection of a specific mode in the memory device includes: The selection of the specific mode is triggered by the mode register.

16. The method of claim 14 or 15, wherein, Receiving the second signal generated by the memory die of the reference memory device includes: The second signal is received from the memory device using a threshold number of input / output (IO) pins coupled to the memory.

17. The method of claim 16, wherein, Receiving the second signal generated by the memory die of the reference memory device includes: Receive a success or failure indicator, the indicator being specific to instructions to be executed by the processor unit; and Receive an identifier coupled to the mode register of the processor unit used to execute the instructions.

18. The method as claimed in any of the preceding claims, wherein, The computational operation triggered to be performed in the memory device is part of the computation for a machine learning inference workload.

19. A system-on-a-chip (SoC), comprising: Processing device; as well as A non-transitory machine-readable storage device for storing instructions that can be executed by the processing device to cause the execution of operations, including: A first signal is generated at the SoC; Based on the first signal, the execution of a computational operation is triggered in the memory device coupled to the SoC; After the execution of the computational operation is triggered in the memory, a second signal generated by referencing the memory die of the memory device is received from the memory device; and The state of the computation operation is determined based on the second signal.

20. A non-transitory machine-readable storage device storing instructions executable by a processing device to cause the execution of operations, the operations including: A first signal is generated at the SoC; Based on the first signal, the execution of a computational operation is triggered in the memory device coupled to the SoC; After the execution of the computation operation is triggered in the memory, a second signal generated by the memory die of the memory device is received from the memory device; as well as The state of the computation operation is determined based on the second signal.

21. A computer-implemented method, comprising: Determine the memory access range at the System-on-Chip (SoC); A write command is generated based on the memory access range; The write command is transmitted to the memory device using a control channel that couples the memory device to the SoC. as well as Access control is configured at the memory device based on the memory access range specified by the write command. The access control restricts the memory-in-memory (PiM) computing elements of the memory device to access only specific memory portions of the memory device.

22. The method of claim 21, wherein, Configuring access control at the memory device includes: Configure row-level memory access constraints at the memory device. The row-level memory access constraint restricts the PiM computing element to accessing a specific range of rows at the memory device.

23. The method of claim 21 or 22, wherein, Configuring access control at the memory device includes: Based on the write command, the access control is configured by writing the memory access range into the configuration space of the PiM architecture in the memory device.

24. The method of any one of claims 21 to 23, wherein, Configuring access control at the memory device includes: The memory portion of the memory device is defined as the only memory portion that the PiM computing element of the memory device can access.

25. The method of any one of claims 21 to 24, wherein, The memory access range includes the physical address range, and the method includes: Write the physical address range to the range register in the memory controller of the SoC; and The physical address range is converted into row, column, and group addresses of the memory device.

26. The method of claim 25, wherein, Generating the write command includes: The write command is generated using the memory controller of the SoC.

27. The method of any one of claims 21 to 26, wherein, Determining the memory access range includes: The memory access range is determined using a trusted agent of the central processing unit (CPU) on the system-on-chip (SoC).

28. The method of any one of claims 21 to 27, wherein, The trusted proxy for the CPU includes: The CPU core; and The CPU management program.

29. A system-on-a-chip (SoC), comprising: Processing device; as well as A non-transitory machine-readable storage device for storing instructions that can be executed by the processing device to cause the execution of operations, including: Determine the memory access range at the System-on-Chip (SoC); A write command is generated based on the memory access range; The write command is transmitted to the memory device using a control channel that couples the memory device to the SoC; and Access control is configured at the memory device based on the memory access range specified by the write command. The access control restricts the memory-in-memory (PiM) computing elements of the memory device to access only specific memory portions of the memory device.

30. A non-transitory machine-readable storage device for storing instructions executable by a processing device to cause the execution of operations, the operations including: Determine the memory access range at the System-on-Chip (SoC); A write command is generated based on the memory access range; The write command is transmitted to the memory device using a control channel that couples the memory device to the SoC. as well as Access control is configured at the memory device based on the memory access range specified by the write command. The access control restricts the memory-in-memory (PiM) computing elements of the memory device to access only specific memory portions of the memory device.

31. A computer-implemented method, comprising: At the system-on-chip (SoC) level, determine the refresh requirements for refreshing rows of memory devices coupled to the SoC; Based on the refresh requirements, one or more refresh commands are generated at the SoC; The one or more refresh commands are transmitted to multiple memory processing (PiM) computing elements of the memory device coupled to the SoC; as well as Based on the one or more refresh commands transmitted to the plurality of PiM computing elements, the refresh of memory cells at each of the rows of the memory device is actively triggered.

32. The method of claim 31, wherein, Actively triggering a refresh of the memory cell at each of the rows of the memory device includes: The refresh of the memory cell is actively triggered at a frequency exceeding the minimum refresh requirement for refreshing rows of the memory device.

33. The method of claim 32, wherein, The memory cell at each of the rows is actively refreshed using the plurality of PiM computing elements of the memory device at a frequency exceeding the minimum refresh requirement.

34. The method according to any one of claims 31 to 33, wherein, Transmitting the one or more refresh command signals includes: The one or more refresh command signals are transmitted within one or more clock cycles generated by the memory controller of the SoC.

35. A system-on-a-chip (SoC), comprising: Processing device; as well as A non-transitory machine-readable storage device for storing instructions that can be executed by the processing device to cause the execution of operations, including: At the system-on-chip (SoC) level, determine the refresh requirements for refreshing rows of memory devices coupled to the SoC; Based on the refresh requirements, one or more refresh commands are generated at the SoC; Transmit the one or more refresh commands to multiple memory processing (PiM) computing elements coupled to the memory device of the SoC; and Based on the one or more refresh commands transmitted to the plurality of PiM computing elements, the refresh of memory cells at each of the rows of the memory device is actively triggered.

Images