Memory bank group data aggregation for improving in-memory processor data access performance

Abstract

Methods, systems, and apparatus for improving PiM data access performance using cross-memory bank data aggregation are disclosed, including computation instructions / programs encoded on a non-transitory computer-readable medium. A system determines an interleaved configuration between a PiM block in a memory device of an integrated circuit and a memory bank outside the PiM block. Based on the interleaved configuration and timing constraints for executing successive read commands to different memory banks outside the PiM block of the memory device, the system provides corresponding operands from the memory banks to the PiM block. The system triggers the execution of computations on memory-constrained workloads at the memory device. These computations are performed at the PiM block using the corresponding operands provided from the memory banks.

Description

Technical Field

[0001] This specification generally relates to an architecture for performing computations in a memory device coupled to a system-on-a-chip (“SoC”). Background Technology

[0002] This specification generally relates to memory devices used to perform calculations.

[0003] Modern computing systems typically contain a wide variety of computing processing units, each offering different computing capabilities and trade-offs. Efficient execution of a given computational job usually involves parsing the computation into meaningful workloads or subtasks, which are then mapped to the available processor cores of the computing system. This parsing and mapping can be based on suitability criteria such as processor capacity, performance, and power consumption. This overall process of allocating portions of the computation to appropriate processor resources is often referred to as heterogeneous computing.

[0004] At least one processing unit of the computing system may be a block of intellectual property (“IP block”) representing a corresponding portion of the computational operations performed for different multimedia workloads. Example use cases may involve processing image or audio data captured, respectively, by a camera or microphone on a mobile device. The SoC may use heterogeneous computing operations to process input samples derived from image data, audio data, or both. Example steps in such heterogeneous computing operations may include providing input samples from image data to a host processor of the SoC, such as a neural network processor or a machine learning (ML) engine, to generate inference output. Summary of the Invention

[0005] This specification describes hardware and software technologies that utilize the concepts of memory bank groups and cross-memory bank data aggregation to improve or accelerate data access performance in in-memory processor architectures (“PiM architectures”) for memory-constrained and compute-constrained workloads in memory devices. The PiM architecture defines one or more PiM blocks of a memory device, and each PiM block includes computational elements such as processor units, mode registers, and one or more arithmetic logic units (ALUs). For example, a PiM block may include discrete processors, processor units, register devices, buffers, etc., that cooperate to form one or more PiM computational elements.

[0006] This technology includes generating data and control signaling at a system-on-chip (“SoC”) that communicates with the PiM architecture and memory device via the SoC’s memory controller. The data and control signaling generated on the SoC is passed to and processed by computing elements within a given PiM block of the PiM architecture. For example, the data and control signaling is routed to and processed by computing elements of the PiM architecture to trigger the execution of specific data processing and computational operations based on memory bank groups and cross-memory bank data aggregations performed in the memory device. Control signals can also be generated locally on the memory device, which may be external to the SoC.

[0007] This PiM architecture is integrated into example memory devices such as Dynamic Random Access Memory (DRAM) or Double Data Rate (DDR) Synchronous DRAM (SDRAM). Through this PiM architecture and its multiple PiM computing elements, the memory device is configured for in-memory processor (“PiM”) operations and in-memory computation operations (“CiM”). The memory device can be external (or internal) to the SoC. In some implementations, the SoC is integrated (or co-located) with the memory device, for example, as disparate circuit dies co-located in a single integrated circuit package.

[0008] Existing design methods for memory devices co-locate the PiM block with the memory bank of the memory bank group, which limits data access performance and the speed or frequency of executing read commands. For example, co-locating the PiM block with the memory bank of the memory bank group constrains the data access frequency, speed, and bandwidth of the PiM block based on long-latency timing constraints (e.g., TCCD_L) to execute successive read commands on the memory bank of the same memory bank group. Compared to these existing methods, the technology disclosed in this specification provides a unique hardware architecture that places one or more PiM blocks outside the memory bank group. This unique placement of the PiM block relative to the memory bank group allows for improved and / or accelerated data access performance by performing data access operations based on short-latency timing constraints (e.g., TCCD_S).

[0009] For example, when the PiM block is placed outside the memory bank group, a short-latency timing constraint is used to achieve faster data access speeds and a higher frequency of executing consecutive read commands. This is because, compared to a long-latency constraint, the short-latency constraint imposes a shorter time delay on consecutive read commands across memory banks, while the long-latency constraint imposes a longer time delay on consecutive read commands across memory banks. For example, the time delay imposed by the short-latency constraint can be half the time delay imposed by the long-latency constraint.

[0010] The hardware architecture includes a data interconnect that: i) couples one or more PiM blocks of a memory device to one or more memory banks; and ii) allows data to be concurrently routed from two or more memory banks of a memory bank group to the one or more PiM blocks. The unique hardware architecture also includes interleaving configuration logic that manages the routing of data to the PiM block based on a specific interleaving configuration. This interleaving configuration logic is software programmable and can be dynamically programmed to implement a series of configurations for interleaving portions of data or operands routed to the PiM block of the memory device.

[0011] This SoC can receive requests to execute memory-constrained ML workloads and cooperate with the memory device to perform computations locally on that memory device, thereby executing some (or all) of the memory-constrained workload within the memory device. For example, a memory-constrained workload may include computational tasks in which the overall execution time of the task is primarily determined by memory access speed. This means that the process executing the workload may spend most of its time reading and writing data from memory, rather than performing complex computations. This allows memory bandwidth and optimized data access speeds to contribute to improved workload performance.

[0012] The SoC can also generate control signals to trigger the execution of computations for the memory-constrained workload at a specific PiM block using data values ​​or operands accessed and / or provided from dissimilar memory banks at a higher frequency based on the short-latency timing constraint. Additionally, the SoC can generate control signals to define specific interleaving configurations, such that specific data values ​​are routed to a given PiM block of the memory device. In this way, the memory device leverages this unique hardware architecture to enhance or accelerate data access performance at the memory device, which also improves the overall timing used to execute the memory-constrained workload.

[0013] One aspect of the subject matter described in this specification can be embodied in a method implemented using an integrated circuit having a System-on-a-Chip (SoC) and a memory device coupled to the SoC. The method includes: determining an interleaving configuration between a PiM block in the memory device and a group of memory banks outside the PiM block; providing corresponding operands from the memory banks to the PiM block based on the interleaving configuration and timing constraints for executing successive read commands on the dissimilar memory banks outside the PiM block in the memory device; and triggering the execution of a computation of a memory-constrained workload, wherein the computation is performed at the PiM block using the corresponding operands from the memory banks.

[0014] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, providing a corresponding operand from the memory bank group includes generating one or more read commands to read the corresponding operand from the memory bank group; reading data representing the corresponding operand from a different memory bank group based on the one or more read commands; and providing the corresponding operand to the PiM block in response to reading the data representing the corresponding operand.

[0015] Providing operands from memory banks includes: providing a first operand from a first memory bank to a PiM block based on the interleaving configuration; and providing a second operand from a second distinct memory bank to the PiM block based on the interleaving configuration and the timing constraint. The timing constraint may be the minimum number of clock cycles between executing consecutive read commands on the first memory bank and the second distinct memory bank.

[0016] In some implementations, the timing constraint includes fewer clock cycles than a second distinct timing constraint, which is the minimum number of clock cycles between executing successive read commands on different memory banks in any bank group of the memory device. Providing corresponding operands from the bank group includes: generating a control signal that causes data to be concurrently transferred from two or more dissimilar memory banks of the first bank group; and concurrently transferring the data from two or more dissimilar memory banks of the first bank group based on the control signal.

[0017] In some implementations, concurrently transferring data from two or more dissimilar memory banks of the first memory bank group includes using a data interconnect between the first memory bank group and a PiM block outside the first memory bank group to concurrently transfer the data. In some implementations, the bandwidth of the data interconnect is sized and configured to allow a threshold number of bytes to be concurrently routed from two or more dissimilar memory banks of the first memory bank group.

[0018] Another aspect of the subject matter described in this specification can be embodied in methods implemented using integrated circuits including a SoC and a memory device coupled to the SoC. The method includes: determining a mapping of resources in the memory device to one or more PiM blocks of the memory device's PiM architecture; and, based on the mapping, generating control signals indicating an interleaved configuration between the PiM blocks and one or more resources in the memory device. The method includes: providing operands from a first resource of the memory device to the PiM block based on the interleaved configuration and a first timing constraint for performing sequential read commands on disparate resources of the memory device; and triggering the execution of a computation of a memory-constrained workload, wherein the computation is performed at the PiM block using operands from the first resource.

[0019] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, determining the resource mapping includes using the SoC's PiM resource manager at the SoC to determine the resource mapping. Generating control signals includes generating control signals at the SoC based on the resource mapping determined using the SoC's PiM resource manager. Determining the resource mapping may include using the interleaved configuration logic of the memory device at the memory device to determine the resource mapping.

[0020] In some implementations, the interleaving configuration logic is software-programmable. Generating control signals includes generating control signals at the memory device based on a resource mapping determined using the interleaving configuration logic of the memory device. The method may include: transmitting control signals within the memory device; and configuring the interleaving configuration at the memory device based on the control signals.

[0021] Another aspect of the subject matter described in this specification can be embodied in an integrated circuit of a memory device. The integrated circuit includes: a PiM block; at least two disparate memory bank groups outside the PiM block, each memory bank group having two or more disparate memory banks; and a data interconnect configured to: i) couple the PiM block to at least a first memory bank group of the two disparate memory bank groups; and ii) permit concurrent routing of data from two or more disparate memory banks of the first memory bank group to the PiM block. The integrated circuit also includes interleaving configuration logic configured to manage the routing of data to the PiM block based on an interleaving configuration.

[0022] Other implementations of this and other aspects include corresponding systems, devices, and computer programs configured to perform actions coded on computer storage devices for these methods. A system of one or more computers may be configured in this way by means of software, firmware, hardware, or combinations thereof installed on the system that cause the system to perform actions during operation. One or more computer programs may be configured in this way by means of instructions having instructions that cause a data processing device to perform actions when executed by the device.

[0023] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0024] Figure 1 This is a block diagram of an example computing system with at least one SoC.

[0025] Figure 2 An example PiM architecture with corresponding computing elements is shown.

[0026] Figure 3 The first example of a PiM block and its corresponding memory bank group is shown.

[0027] Figure 4 A second example of a PiM block and its corresponding memory bank group is shown.

[0028] Figure 5A and Figure 5B Each example PiM architecture is shown to improve PiM data access performance.

[0029] Figure 6 This is an example procedure for improving PiM data access performance.

[0030] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0031] Figure 1 This is a block diagram of an example computing system 100, which includes a system-on-a-chip 102 (“SoC 102”). SoC 102 includes a central processing unit 104 (“CPU 104”), a memory controller 105, shared memory 106 (“memory 106”), a resource manager 108, and IP / circuit blocks 110. In some implementations, system 100 may include multiple SoCs, and any description of SoC 102 will equally apply to each of the multiple SoCs that may be included at system 100.

[0032] CPU 104 can be a general-purpose CPU (e.g., a single-core or multi-core CPU). CPU 104 generates one or more indicators, such as an app launch indicator or a function call triggered in response to the execution or launch of an application on a user device. For example, the application could be a camera application that uses an imaging sensor to generate image data, or a game application that requires significant memory and graphics processing resources to render the game's graphical content. CPU 104 also generates one or more application values, such as pixel values ​​or frame rates. Application values ​​may be associated with function calls, may describe events that occur during the execution of the application, or both.

[0033] Memory 106 is system memory, shared memory, or both. Figure 1In the example, memory 106 is depicted as being outside circuit block 110. However, memory 106 may include the following portions of memory: i) a portion specific to circuit block 110, ii) a portion outside circuit block 110, or iii) both. Memory 106 may be the random access memory of SoC 102, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0034] In some implementations, aspects of memory 106 are configured as shared scratchpad memory, which supports parallel access to its memory resources by two or more processors of circuit 110. Memory 106 may also include various other types of memory, such as high-bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0035] Resource Manager 108 is implemented in both hardware and software. Aspects of Resource Manager 108 may also be implemented as firmware of SoC 102, or firmware of a device of SoC 102, such as a DRAM memory device or CPU 104. Resource Manager 108 is an in-memory processor (PiM) resource manager (“PiM Resource Manager 108”), which includes control logic implemented in hardware, software, or both. For example, PiM Resource Manager 108 may include resources implemented in hardware, such as flip-flops, registers, and buffers, as well as control logic implemented in software (e.g., programming code).

[0036] Circuit block 110 typically includes various IP devices, such as processors, processor cores, or dedicated processing devices. For example, circuit block 110 may include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. Circuit block 110 is alternatively referred to as IP block 110, where an IP block may include one or more proprietary hardware elements. For example, each of ISP 112, TPU 114, DSP 116, and GPU 118 may be a corresponding proprietary IP block (or IP device) from a particular entity or device manufacturer.

[0037] One or more aspects of the PiM Resource Manager 108 can be implemented as software routines (or modules) of the CPU 104 that utilize one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from the system memory of the SoC 102, such as memory 106. In some implementations, each processor of the SoC 102 (e.g., ISP 112, DSP 116, TPU 114, GPU 118) includes multiple cores, and the CPU 104 and / or the PiM Resource Manager 108 can generate control signaling 124 to manage memory-intensive computational operations and distribute these operations across memory devices 122 (e.g., DRAM), thereby minimizing the processing load at each core of the processor. The control signaling 124 is routed at the system 100 using the example bus 120 of the SoC 102. Control signaling 124 may include commands, requests, data, instructions, or combinations thereof.

[0038] PiM Resource Manager 108 collaborates with CPU 104, memory controller 105, and memory controller 107 to dynamically control and manage one or more in-memory compute (CIM) operations. In some implementations, CIM operations are performed at SoC 102 to support heterogeneous compute operations between two or more processing units included in IP block 110, CPU 104, or both. More specifically, PiM Resource Manager 108 is configured to generate control signaling 124 and use one or more discrete signal values ​​of control signaling 124 to manage and elevate data access operations at memory device 122.

[0039] System 100 includes an example memory device 122. Memory device 122 may include multiple memory dies. For example, memory device 122 may include N memory dies, where N is an integer greater than 1. In the example shown, memory device 122 may be dynamic random access memory (DRAM) or double data rate (DDR) synchronous DRAM (SDRAM). Memory device 122 is configured to perform or support various types of PiM operations, CiM operations, and near-memory computation operations (“MnC operations”). Memory device 122 performs or supports these operations using its multiple PiM computation elements, which are referenced below. Figures 2 to 4 Describe it.

[0040] SoC 102 collaborates with memory device 122 to perform computations across one or more banks of memory on memory device 122. The computations can be used for operations or workloads involving one or more processors at IP block 110. Alternatively, the computations can be used for heterogeneous operations across multiple processors, multiple IP blocks 110, or both. In at least one example, memory device 122 may be external to SoC 102, while in another example, memory device 122 may be internal to SoC 102.

[0041] exist Figure 1 In the example, system 100 and SoC 102 are integrated circuits for example user / client device 130, consumer electronic device, or mobile device, each of which may include items such as smartphone 130a, tablet 130b, laptop 130c, smartwatch, or wearable device 130d. Device 130 may also include other items such as netbook, smart speaker, or mobile computer. In some implementations, system 100 and SoC 102 are integrated circuits for desktop computers, web servers, or associated cloud-based assets.

[0042] Figure 2 An example in-memory processor (PiM) architecture 200 is shown for improving PiM data access performance based on control signals generated using SoC 102, memory device 122, or both. As described above, memory device 122 may include N memory dies, where N is an integer greater than 1.

[0043] exist Figure 2 In the example, memory device 122 includes a first memory die 1, memory die 210-1, having a first memory bank group having a plurality of memory banks, wherein each memory bank includes one or more memory arrays; and a second memory die 2, memory die 210-2, having a second memory bank group having a plurality of memory banks, wherein each memory bank includes one or more memory arrays. In some implementations, PiM architecture 200 includes a plurality of memory bank groups, a plurality of memory dies, or both. For example, a single memory die may include a plurality of memory bank groups, and / or the plurality of memory bank groups may be distributed across a plurality of memory dies.

[0044] PiM architecture 200 includes multiple PiM blocks, each comprising multiple computing elements. For example, a first PiM block of PiM architecture 200 includes a mode register 204-1 and a processing unit 206-1, while a second distinct PiM block of PiM architecture 200 includes a mode register 204-2 and a processing unit 206-2. Each processing unit 206-1, 206-2 may include a processor, a processor unit, or a processor core, such as a CPU. Each processing unit 206-1, 206-2 may also include example computing units, such as an arithmetic logic unit (ALU) or a multiply-accumulate unit (MAC). The PiM architecture 200 disclosed herein, as well as other PiM architectures, may include multiple control and status registers (CSRs), which represent auxiliary registers used for reading status and changing configuration and operating modes within the PiM architecture.

[0045] In some implementations, the PiM architecture 200 is included in the memory device 122 as a plurality of discrete integrated circuits, wherein each integrated circuit is local to a given memory die (e.g., die 1 and die 2) and interacts or communicates with the memory cell array at that memory die. For example, the PiM architecture 200 may include computing elements that are replicated and distributed across each memory die in the memory device 122. In some other implementations, the PiM architecture 200 is included in the memory device 122 as a single integrated circuit that interacts or communicates with each memory die of the memory device 122—including the memory cell array at each memory die 210-1, 210-2.

[0046] To improve PiM data access performance as described in this specification, PiM blocks or processing units in PiM architecture 200 are located within memory device 122, but outside a segment of memory device 122 that includes a bank group. This segment can be defined as a discrete memory die, or it can be defined in some other way (e.g., a portion of a memory die). Regardless of the hardware configuration or layout of PiM architecture 200, the PiM blocks are fully located outside the bank group, allowing the PiM blocks to communicate with the bank group based on specific timing constraints that can be used to improve PiM data access performance through cross-bank group data aggregation.

[0047] In some implementations, the PiM architecture 200 can be fully located outside or separated from the memory bank group through a combination of physical distance, communication protocols, or data connections. This allows different PiM blocks of the PiM architecture 200 to perform data access operations across different memory banks of the memory device 122. Data channels / interconnects and corresponding interleaved configuration logic can be established between the various memory arrays of the memory bank and the corresponding processing units and / or mode registers of the PiM blocks. See below for reference. Figure 5A and Figure 5B The example describes additional details of this implementation.

[0048] PiM operations can include standard CPU functions, while CiM and MnC operations can include standard arithmetic operations, such as computations typically performed by the ALU or MAC. CiM and MnC operations can also include computational functions of the TPU 114, such as multiplication and addition operations for matrix mathematics, vector computation, linear algebra, and dot-matrix accumulation. In some implementations, each of the PiM, CiM, and MnC operations is performed to support machine learning computations, neural network computations, or both.

[0049] PiM architecture 200 may include registers or other portions of memory for storing error data for the corresponding memory die or memory bank. For example, the error data may describe an error that occurred during a computational operation at the corresponding PiM block of memory device 122. In some implementations, other portions of registers or memory are used to store configuration information or associated instructions for configuring various aspects of the PiM block or the corresponding memory die, memory bank, or a combination thereof. For example, mode registers 204-1 and 204-2 may be used to control or trigger the selection of specific modes in the PiM architecture, such as error capture mode, interleaved configuration mode, etc.

[0050] In some implementations, a specific mode is selected based on the bit values ​​of mode registers 204-1 and 204-2. For example, a single bit (or a sequence of bits) can be defined for use in the mode registers to trigger or select one or more interleaved configuration modes. The bit values ​​of mode registers 204-1 and 204-2 are set by commands / instructions received from the SoC 102, local commands / instructions executed by the processing unit of the PiM architecture 200, or both. For example, the value of the PiM mode register can be programmed using a mode register write command (MRW), while other PiM registers, such as the PiM CSR, can be programmed using a PiM control register write command. A subset of bits in the example control signals can be used to trigger certain interleaved configuration modes and memory bank group (or cross-memory bank) data aggregation modes / settings of the PiM architecture 200.

[0051] The control signal may be a dedicated PiM config_control signal generated at SoC 102. Alternatively, the control signal may be a read request signal generated by the memory controller 105 of SoC 102. For example, in addition to its primary purpose of requesting, reading, or obtaining data from memory device 122, the read request signal (and its associated bit data structure) may also be reused for the secondary purpose of providing a trigger for the execution of computational operations in the PiM module of memory device 122. Based on control signaling, specific interleaved configuration modes or data aggregation modes are triggered and / or configured in PiM architecture 200. In some cases, such triggering or configuration of certain modes in PiM architecture 200 occurs when a command from SoC 102 triggers the batch execution of multiple PiM instructions in memory device 122.

[0052] The frequency of memory device 122 can be measured in megahertz (MHz) and indicates the number of cycles that memory device 122 (e.g., DRAM or SDRAM) can perform in one second. This number of cycles represents the number of data transfers that can occur, for example, between PiM blocks and memory banks (or memory banks) of memory device 122 in a given time frame. Generally, higher DRAM frequencies result in faster data access and better system performance.

[0053] For clarity, memory device 122 may have an example structure of multiple memory banks, wherein each memory bank includes one or more memory arrays, and each memory array includes a set of memory cells arranged, for example, in a row or column format. Typically, a memory cell may be the smallest unit of memory storing, for example, a single bit of data, while a memory array may be a collection of multiple memory cells organized together to store a larger amount of data. In some implementations, a memory array may be defined as a two-dimensional array of memory cells, wherein data is stored and subsequently retrieved or read and routed to PiM blocks 212-1, 212-2 for computational processing at the PiM blocks using an arithmetic circuit system of PiM blocks 212-1, 212-2.

[0054] Memory banks are arranged into different memory bank groups, referred to herein as memory bank groups. In some examples, memory cells, memory arrays, memory banks, memory bank groups, and example structures of memories comprising multiple memory bank groups are implemented across one or more circuit dies. Each memory bank group may represent a separate memory entity / cell configured such that row or column cycle memory access operations can be performed within that memory bank group without affecting operations occurring in another memory bank group.

[0055] These cells, arrays, banks, and groups of memory device 122 represent resources (or memory resources) of memory device 122. The control logic of system 100 is configured to determine the mapping of resources in memory device 122 to one or more PiM blocks of the memory device's PiM architecture. Based on this mapping, the configuration logic of system 100 can generate control signals that indicate the interleaved configuration between PiM blocks and one or more resources in the memory device. This is described in more detail below.

[0056] Using the disclosed technique, data access operations are optimized by reading memory cells of the memory bank at a frequency exceeding the read command frequency provided by long-latency timing constraints such as TCCD_L timing constraints. As described above, the innovative technique disclosed in this specification leverages the unique placement of PiM blocks relative to the memory bank entity to improve and / or accelerate data access performance by performing data access operations based on short-latency timing constraints such as TCCD_S. As used in this document, "CCD" can refer to column-to-column latency or column-side command-to-command latency. "_S" stands for "short," and "_L" stands for "long."

[0057] The internal controller of PiM block (1) 212-1 or PiM block (2) (212-2) can execute successive read commands at a frequency based on the clock cycle generated by memory device 122. The internal controller can operate based on a specific clock frequency, such as a 200 or 800 MHz clock or a 1000 MHz clock. Other clock frequencies are also within the scope of this disclosure. In some implementations, the tCCD parameter is defined differently between the same memory bank group and different memory bank groups.

[0058] The internal controller of the PiM architecture can leverage the TCCD_S timing constraint to perform cross-memory bank access at, for example, nanosecond (ns) intervals instead of the slower intervals associated with the TCCD_L timing constraint. For instance, the time delay imposed by the short-latency timing constraint (TCCD_S) can be half the time delay imposed by the long-latency constraint (TCCD_L). Given this, the disclosed technique for cross-memory bank data access utilizes a relation (TCCD_S = ½) The TCCD_L timing is reduced by half, and the TCCD_S timing constraint is used to achieve higher frequency data access operations (relative to TCCD_L). Data access operations can be represented by read command signals triggered by instructions processed at the PiM architecture of memory device 122.

[0059] In some implementations, memory device 122 includes a data rate of 6400 megabits per second (Mbps), which represents an example amount of data that memory device 122 can transfer per second. In this example, a clock cycle (t) can be used. CK The delay imposed by the timing constraints TCCD_S and TCCD_L is defined using ). For example, the TCCD_S constraint can impose a four (4t) delay. CK The TCCD_L constraint is delayed, while TCCD_S imposes a constraint greater than that of TCCD_S. For example, a TCCD_L constraint could be an octet (8t) constraint. CK Periodic delay period. In this example, if the PiM block operates based on a 200MHz clock, then the data rate is: 6400Mbps → WCK: ½ data rate (6400MHz) → CK: ¼ WCK (3200MHz) → t CK : 1.25ns → 4 tCK (4 The time interval is 1.25ns, and 5ns → 200MHz. For example, when staying within the same memory bank at 1,600 Mbps, the tCCD_L constraint requires more than four clocks. In these examples, CK represents the clock and WCK represents the write clock, each corresponding to the clock frequency of memory device 122.

[0060] Figure 3 The first example PiM architecture 300 is shown. Figure 3 In the example, architecture 300 includes memory bank groups 302 and 304, each including at least two memory banks (310-1 and 310-2, and 312-1 and 312-2, respectively). Architecture 300 also includes at least two PiM blocks 306-1 and 306-2. PiM architecture 300 represents a design option where PiM blocks 306-1 and 306-2 are constrained to operate within a specific memory bank group and perform read / write operations on that specific memory bank group. For example, as shown, PiM block 306-1 is located within memory bank group 302 and is constrained to perform its data access operations within memory bank group 302, while PiM block 306-2 is located within memory bank group 304 and is constrained to perform its data access operations within memory bank group 304.

[0061] In PiM architecture 300, PiM blocks read data from each memory bank one array / bank at a time. For example, during operation, PiM block 212-1 can be instructed to read data from each of memory banks 310-1 and 310-2 within memory bank group 302. In this example operation, PiM block 306-1 performs its read operation based on timing constraints and corresponding frequencies. For example, the timing constraints could be based on TCCD_L, as defined by the Joint Electronic Devices Engineering Committee (JEDEC) specification, for performing sequential read operations on memory banks of the memory bank group. Operating under TCCD_L, the PiM block first reads 32 bytes (32B) of data from memory bank 310-1, and once completed, reads 32B of data from memory bank 310-2. In this architecture, similar operations occur in memory bank group 304 using PiM block 306-2 and memory banks 312-1 and 312-2.

[0062] Figure 4 A second example, the PiM architecture 400, is shown. Figure 4 In the example, architecture 400 includes memory bank groups 402 and 404, each of which includes at least two memory banks (310-1 and 310-2, and 312-1 and 312-2, respectively). Figure 3 In contrast, architecture 400 includes four PiM blocks 206-1, 206-2, 206-3 and 206-4—two of these PiM blocks operate within memory bank 402 and two operate within memory bank 404.

[0063] Operation of Architecture 400 Figure 3 Similar to the architecture 300, the PiM block reads 32B of data from the memory bank based on the TCCD_L timing constraint and the associated operating frequency. Figure 3 Conversely, since each memory bank group in architecture 400 includes two PiM blocks, data can be read from multiple memory banks simultaneously. For example, in memory bank group 402, PiM block 212-1 can read data from memory bank 310-1, while PiM block 212-2 reads data from memory bank 310-2. Therefore, assuming Figure 3 and Figure 4 The operation is the same as in architecture 300, but it can be performed faster in architecture 400 because multiple memory banks can be read simultaneously by a larger number of PiM blocks. However, when comparing architecture 400 to architecture 300, the additional PiM blocks 206-2 and 206-4 in architecture 400 result in a larger circuit coverage area.

[0064] PiM Architecture 400 can be an extension of PiM Architecture 300, but it is a more limited design compared to PiM Architecture 200. Figure 4In the example, SoC 102 communicates with computing elements of PiM architecture 400 to configure data access at memory device 122. For example, computing elements such as mode registers 204-1, 204-2 and corresponding processing units 206-1, 206-2 are used to trigger or define data access configurations for executing read commands on memory cells along rows of each DRAM bank of memory die 1 or die 2.

[0065] Figure 5A and Figure 5B Each example PiM architecture is shown to improve PiM data access performance. Figure 5A An example of PiM architecture 500 performing operations is shown, where each PiM block reads data from multiple memory arrays / memory banks within the same memory bank group, while Figure 5B The “cross-bank” configuration is shown, in which PiM blocks 212-1 and 212-2 read data from memory banks 310-1 and 312-1 in different memory bank groups 502 and 504, respectively.

[0066] exist Figure 5A and Figure 5B In the example, architecture 500 includes two PiM blocks 212-1 and 212-2, two memory bank groups 502 and 504, each with two memory banks (310-1, 310-2, 312-1, and 312-2), interleaved configuration logic 508, and a data channel 510. Notably, in architecture 500, PiM blocks 212-1 and 212-2 are located outside of memory bank groups 502 and 504. Although only a specific number of PiM blocks, memory bank groups, and memory banks are depicted, architecture 500 is scalable and can be adapted to operate with different arrangements and numbers of these components.

[0067] As mentioned above Figure 3 and Figure 4 As discussed herein, memory banks 502 and 504 operate at a set frequency, which corresponds, for example, to a long-delay timing constraint such as the TCCD_L timing constraint of the JEDEC specification. By positioning or placing PiM blocks 212-1 and 212-2 outside of memory banks 502 and 504, the PiM blocks can operate at a higher frequency (e.g., twice the frequency of TCCD_S—TCCD_L in the JEDEC specification). This higher operating frequency allows PiM blocks 212-1 and 212-2 to read data from the memory array of memory banks 310, 312 at a faster rate—e.g., twice the normal rate when the PiM blocks are located within the memory banks.

[0068] Figure 5AExample operations are shown where PiM blocks 212-1 and 212-2 read data from multiple memory banks within the same memory bank groups 502 and 504. For example... Figure 5A As shown, PiM block 212-1 reads data from memory banks 310-1 and 310-2 in memory bank group 504. A similar operation occurs in PiM block 212-2, which reads data from memory banks 312-1 and 312-2 in memory bank group 502.

[0069] Given that PiM blocks 212-1 and 212-2 are with Figure 3 and Figure 4 Compared to twice the data reading rate of the previous method, the data channel 504A can be adjusted to accommodate higher data rates. For example, as... Figure 5A As shown, for each memory bank in the memory bank group, data channel 510 is sized to 32 bytes (32B) (64 bytes (64B) in total). The size of data channel 510 can be appropriately scaled depending on the number of memory banks in a particular memory bank group that will be read simultaneously.

[0070] Additionally, architecture 500 includes interleaving configuration logic 508, which controls the operation of PiM blocks 212-1 and 212-2 to resolve data dependencies between memory banks or across memory bank groups. For example, if data within memory bank 310-1 depends on data from memory bank 310-2, interleaving configuration logic 508 can cause PiM block 212-1 to read data from memory bank 310-2 first. Similarly, if data from memory bank group 502 is needed to read data from memory bank group 504, interleaving configuration logic 508 can be configured to cause PiM blocks 212-1 and 212-2 to read data stored in memory bank group 502 first. The operation of interleaving configuration logic 508 can be dynamic and adapts to data dependency requirements as operations are assigned to PiM blocks 212-1 and 212-2.

[0071] Figure 5B An example operation (“cross-bank” configuration) is shown in which PiM blocks 212-1 and 212-2 read data from multiple memory banks within different memory bank groups 502 and 504. In this example operation, PiM block 212-1 reads data from memory bank 310-1 of memory bank group 502 and memory bank 312-1 of memory bank group 504. A similar operation occurs on PiM 212-2. As described above... Figure 5A As described, the data channel 504B is resized to account for the number of memory banks that will be read simultaneously (e.g., 32 bytes per memory bank). Additionally, as referenced... Figure 5AThe interleaved configuration logic 508 discussed addresses any data dependency issues that arise during PiM operation.

[0072] because Figure 5A and Figure 5B The PiM blocks 212-1 and 212-2 shown are located outside memory bank groups 502 and 504, therefore the operating frequency limit for data access within the same memory bank group does not apply (e.g., TCCD_L of the JEDEC specification). Therefore, a faster operating frequency (e.g., TCCD_S) can be used, which provides improved processing performance for the same physical coverage area. For example, assuming the same operation, and TCCD_S is twice as fast as TCCD_L, Figure 4 The operation shown can be used Figure 5A or Figure 5B Architecture 500 was completed in the same amount of time. Notably, Architecture 500 requires two fewer PiM blocks than Architecture 400, and therefore can achieve the same performance with a smaller physical coverage area.

[0073] Figure 6 This is an example process for improving PiM data access performance in memory device 122. Process 600 is implemented or performed at system 100 using at least the aforementioned SoC 102 and memory device 122. Therefore, the description of process 600 will refer to the computing resources of system 100 mentioned above. In some examples, the steps or actions of process 600 are implemented by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and may be executed by one or more of the processors or other resources described in this specification.

[0074] Referring again to process 600, system 100 determines an interleaved configuration (602) between an in-memory processor (“PiM”) block within a memory device and at least two distinct memory bank groups outside that PiM block. For example, CPU 104 or PiM resource manager 108 may dynamically determine the interleaved configuration based on the type of PiM operation or CiM operation being performed at memory device 122. CPU 104 or PiM resource manager 108 may dynamically determine the interleaved configuration based on the type of memory device 122 coupled to SoC 102, the number of memory banks in each memory bank group of memory device 122, the layout of data values ​​or operands across memory bank groups of memory device 122, or a combination thereof. In some implementations, memory device 122 includes software-programmable interleaved configuration logic. In these implementations, the software-programmable aspects of the interleaved configuration logic can be used to dynamically configure and / or reconfigure the interleaved configuration at memory device 122.

[0075] SoC 102 can determine the mapping of memory bank groups to PiM blocks for memory device 122. This mapping can be used to define interleaved configurations and data access requirements for rows or columns of memory banks within a specific memory bank group coupled to the SoC. In some implementations, SoC 102 defines the interleaved configuration at memory device 122 by determining the mapping between PiM blocks 212-1, 212-2 and rows or columns of memory banks in memory banks 502, 504 on a specific memory die of memory device 122.

[0076] When data values ​​are transmitted to PiM blocks via data interconnect 510, row-level or column-level mappings can be indicated or referenced by an interleaving configuration that interleaves data values ​​stored at rows or columns of the memory bank between two or more PiM blocks. In some implementations, the bandwidth of the data interconnect 510 is sized and configured to allow concurrent routing of a threshold number of bytes from two or more dissimilar memory banks of the first memory bank group. For example, the threshold number of bytes can be 16B, 32B, 64B, or 128B, where "B" is a byte. In some implementations, the threshold can be greater than or less than 32B.

[0077] System 100 provides operands or data values ​​from different memory banks outside the PiM block to the PiM block based on an interleaved configuration and timing constraints for executing successive read commands on the memory device. For example, memory device 122 can provide data values ​​from two different memory banks, transfer data values ​​via data interconnect 510, and interleave appropriate subsets of data values ​​to PiM blocks 212-1, 212-2 based on the interleaved configuration. In some implementations, SoC 102 uses CPU 104 and / or PiM resource manager 108 to generate control signals and pass them to memory device 122, which define the interleaved configuration and trigger data access operations that cause different memory banks of memory device 122 to provide data values ​​to one or more PiM blocks 212-1, 212-2 based on interleaved configuration logic 508.

[0078] System 100 triggers the execution of computations for a memory-constrained workload (606). As described above, a memory-constrained workload may include computational tasks, where the total execution time of the task is primarily determined by the speed of memory access. In this case, the process workload execution may need to spend a significant amount of time reading and writing data from and to the memory banks of the memory device, rather than performing complex computations. This timing overhead makes memory bandwidth and memory access timing key (e.g., important) factors in the overall performance of the workload.

[0079] These computations are performed at the PiM block using corresponding data values ​​or operands provided to the PiM block from one or more memory banks. For example, SoC 102 can trigger the execution of computations by transmitting control signals (or commands) to memory device 122 using a control channel that couples memory device 122 and the SoC. In some implementations, SoC 102 triggers the execution of computational operations in memory device 122 based on request / command signals generated by the memory controller 105 of SoC 102.

[0080] Request / command signals are passed to the computational elements of PiM blocks 212-1 and 212-2 in memory device 122 to trigger the execution of instructions in the instruction buffers of specific PiM blocks 212-1 and 212-2. For example, the request signal may be passed to the processor of PiM block 212-1, the mode register of PiM block 212-2, or both. The processor (or mode register) uses or reads the request signal to execute computation instructions and generate computation results based on the instructions. The results can be passed back to the memory bank group for storage in the corresponding memory bank within the memory bank group.

[0081] The corresponding steps of process 600 can be performed at a hardware integrated circuit as part of a larger computational operation to generate machine learning (ML) outputs, including outputs for neural network layers implementing one or more ML models. For example, the outputs can be part of computations for generating image processing outputs, speech processing outputs, or image recognition outputs for an ML task or inference workload.

[0082] As indicated above, a portion of the integrated circuit may include a dedicated neural network processor or hardware ML accelerator configured to accelerate computations for generating different types of data processing outputs. In some implementations, memory device 122 performs one or more of PiM, CiM, or MnC operations to support or implement accelerated computations for generating different types of data processing outputs.

[0083] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or to control the operation of the data processing device.

[0084] Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) to generate the artificially generated propagation signals to encode information for transmission to a suitable receiver device for execution by a data processing device. Computer storage media may be machine-readable storage devices, machine-readable storage substrates, random or serial access memory devices, or combinations thereof.

[0085] The term "computing system" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, by way of example, programmable processors, computers, or multiple processors or computers. The device may include dedicated logic circuit systems such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may also include code that creates an execution environment for the computer program involved, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0086] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0087] Computer programs may, but do not need to, correspond to files in a file system. Programs may be stored as a part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple coordinating files, such as files storing one or more modules, subroutines, or code sections. Computer programs may be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected through a communication network.

[0088] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system, and the device can also be implemented as a dedicated logic circuit system, such as an FPGA (Field-Programmable Gate Array), an ASIC (Application-Specific Integrated Circuit), or a GPGPU (General-Purpose Graphics Processing Unit).

[0089] Computers suitable for executing computer programs include, by way of example, those based on general-purpose or special-purpose microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. Some components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both. However, a computer does not need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0090] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0091] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as an LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0092] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0093] A computing system may include clients and servers. Clients and servers are typically located far apart and interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0094] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be characteristic of specific embodiments of a particular invention. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0095] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all shown operations to achieve the desired result. In some contexts, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0096] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method implemented using an integrated circuit, the integrated circuit including a system-on-a-chip (SoC) and a memory device coupled to the SoC, the method comprising: Determine the interleaved configuration between the in-memory processor "PiM" block in the memory device and the memory bank group outside the PiM block; Based on the interleaved configuration and timing constraints for executing successive read commands on disparate memory banks outside the PiM block of the memory device, corresponding operands are provided to the PiM block from each of the multiple memory banks. as well as Trigger the execution of computations for memory-constrained workloads, wherein the computations are performed at the PiM block using the corresponding operands from the memory bank group.

2. The method of claim 1, wherein providing the corresponding operand from the memory bank group comprises: Generate one or more read commands to read the corresponding operands from the memory bank group; Data representing the corresponding operands is read from the disparate memory banks based on the one or more read commands; as well as The corresponding operand is provided to the PiM block in response to reading the data representing the corresponding operand.

3. The method of claim 2, wherein providing the corresponding operand from the memory bank group comprises: Based on the interleaved configuration, a first operand is provided from the first memory bank group to the PiM block; as well as Based on the interleaved configuration and the timing constraints, a second operand is provided from a second distinct memory bank group to the PiM block.

4. The method of claim 3, wherein the timing constraint is the minimum number of clock cycles between executing consecutive read commands on the first memory bank and the second different memory bank.

5. The method of claim 3 or 4, wherein the timing constraint includes fewer clock cycles than a second different timing constraint, the second different timing constraint being the minimum number of clock cycles between executing successive read commands on different memory banks in any bank group of the memory device.

6. The method of claim 5, wherein providing the corresponding operand from the memory bank group comprises: Generate control signals that cause data to be transmitted concurrently from two or more dissimilar memory banks of the first memory bank group; as well as Based on the control signal, the data is transmitted concurrently from the two or more dissimilar memory banks of the first memory bank group.

7. The method of claim 6, wherein concurrently transferring the data from the two or more dissimilar memory banks of the first memory bank group comprises: The data is transmitted concurrently using a data interconnect between the first memory bank and the PiM block outside the first memory bank.

8. The method of claim 7, wherein the bandwidth of the data interconnect is sized and configured to allow a threshold number of bytes to be concurrently routed from the two or more dissimilar memory banks of the first memory bank group.

9. A method implemented using an integrated circuit, said integrated circuit including a system-on-a-chip (SoC) and a memory device, the method comprising: Determine the mapping of resources in the memory device to one or more in-memory processor "PiM" blocks of the memory device; Based on the mapping, control signals are generated that indicate the interleaved configuration between PiM blocks and one or more resources in the memory device; Based on the interleaved configuration and a first timing constraint for executing sequential read commands on disparate resources of the memory device, operands are provided from a first resource of the memory device to the PiM block; as well as Trigger the execution of computations on memory-constrained workloads, wherein the computations are performed at the PiM block using operands from the first resource.

10. The method of claim 9, wherein determining the mapping of the resource comprises: The PiM resource manager of the SoC is used at the SoC to determine the mapping of resources.

11. The method of claim 10, wherein generating the control signal comprises: The control signal is generated at the SoC based on the mapping of resources determined using the SoC's PiM resource manager.

12. The method of claim 9, wherein determining the mapping of the resource comprises: The interleaved configuration logic of the memory device is used at the memory device to determine the mapping of resources.

13. The method of claim 12, wherein the interleaving configuration logic is software programmable.

14. The method of claim 12 or 13, wherein generating the control signal comprises: The control signal is generated at the memory device based on the mapping of resources determined using the interleaved configuration logic of the memory device.

15. The method of any one of claims 9 to 14, further comprising: The control signal is transmitted within the memory device; as well as The interleaving configuration is configured at the memory device based on the control signal.

16. An integrated circuit for a memory device, comprising: The processor "PiM" block within memory; At least two distinct memory bank groups outside the PiM block, each memory bank group comprising two or more distinct memory banks; Data interconnection, wherein the data interconnection is configured as follows: i) Couple the PiM block to at least the first memory bank of the two dissimilar memory banks; as well as ii) Allow data to be concurrently routed from two or more dissimilar memory banks of the first memory bank group to the PiM block; and Interleaved configuration logic, which is configured to manage the routing of data to the PiM block based on interleaved configuration.

Images