Memory access scheduling for parallel computation using a processing-in-memory architecture

Abstract

A method, system, and apparatus for scheduling instructions and memory access in an integrated circuit are disclosed, comprising a computing instruction / program encoded on a non-transient computer-readable medium. The system determines a performance objective for executing an ML workload and schedules instructions for a compute operation to execute said workload in a processor-in-memory ('PiM') block of a memory device based on a first priority. For concurrent memory transactions of both PiM instructions for compute and memory access requests, the system determines a potential violation of said workload's performance objective and resolves said potential violation by adjusting the first priority to exceed a second priority assigned to the memory request. The system satisfies performance objectives for executing the ML workload in the PiM block and for processing the memory request, respectively, by adjusting the first priority to exceed the second priority and adjusting the operating frequency of the memory device and the PiM block.

Description

Background Technology

[0001] This specification generally relates to memory access transactions.

[0002] Modern computing systems typically integrate various compute processing units, each offering different computing capabilities and trade-offs. The efficient execution of a given compute task usually involves parsing computations into meaningful subtasks or workloads that are mapped to the available processor cores of the computing system. Computations can be parsed and mapped based on suitability criteria such as processor capability, performance, and power. Generally, this entire process of allocating portions of computations to appropriate processor resources is referred to as heterogeneous computing.

[0003] At least one processor core of the computing system may be an intellectual property block ('IP block') that executes individual parts of computational operations for different multimedia workloads. Exemplary use cases may involve processing image or voice data captured, respectively, by a camera or microphone on a mobile device, as well as performing computations for generative artificial intelligence ('GenAI') applications. The system may use heterogeneous computing operations to process input samples derived from image data, voice data, text corpora, or a combination thereof. An exemplary step in heterogeneous computing may include processing data associated with the input samples using in-memory processing or a memory device that provides computing capabilities.

[0004] This specification describes hardware and software techniques for memory access scheduling in concurrent use case scenarios using a processing-in-memory ('PiM') architecture of a memory device. Such techniques are implemented using an integrated system that includes data processing resources within the memory device, wherein the data processing resources enable and / or facilitate signal communication with a host device / core within the system-on-chip ('SoC') of the integrated system.

[0005] The memory device may be a dynamic random access memory (DRAM) device having an exemplary PiM architecture. The PiM architecture defines one or more PiM blocks of the memory device, and each PiM block includes computing resources / elements such as a processor unit, a mode register, and one or more arithmetic units (e.g., an arithmetic logic unit (ALU) or related addition and multiplication circuits). For example, a PiM block may include a separate processor, processor unit, register device, buffer, multiplication accumulator cell (MAC), etc., that cooperate to form one or more PiM compute elements.

[0006] PiM blocks are used to execute computations for exemplary workloads, such as machine learning workloads for computing outputs associated with generative artificial intelligence ('GenAI') applications. Machine learning computations may be divided into individual portions allocated between one or more PiM blocks of a memory device and an SoC. The disclosed techniques provide an advantageous Quality of Service ('QoS') method for dynamically changing memory-traffic scheduling policies for memory transactions, wherein memory transactions include i) memory access requests (e.g., read and write requests) and ii) PiM operation commands.

[0007] The SoC can implement this QoS policy to ensure that the ML workload is executed according to given performance requirements, such as a target completion time for executing the workload. For example, to meet performance requirements, the system (100) can automatically prioritize PiM instructions to facilitate the execution of PiM operations on memory devices, which ultimately accelerates the execution of the ML workload. This dynamic QoS scheduling policy can be used to meet respective performance targets for memory access and PiM operations to support the execution of the ML workload.

[0008] One aspect of the subject matter of the invention described herein may be implemented by a method using an integrated circuit comprising a system-on-chip ('SoC') and a memory device coupled to the SoC. The method comprises the steps of: determining a performance objective for executing a machine learning (ML) workload; and executing the ML workload on the integrated circuit by scheduling instructions for compute operations in a processor-in-memory ('PiM') block of the memory device based on a first priority. The method also comprises the steps of: determining a potential violation of the performance objective; resolving the potential violation by adjusting the first priority to exceed a second priority assigned to memory read / write transactions provided to the memory device; and satisfying the performance objective of the ML workload in the PiM block and memory read / write transactions in response to adjusting the first priority to exceed the second priority and adjusting the operating frequency of the memory device and the PiM block.

[0009] Each of these implementations and other implementations may optionally include one or more of the following features. For example, in some implementations, i) instruction and memory read / write transactions are processed concurrently for at least one task of the ML workload; and ii) instruction and memory read / write transactions share an interface communication channel between the SoC and the memory device. In some implementations, the step of executing the ML workload and meeting performance objectives may include the step of performing any remaining compute operations based on an adjusted first priority that prioritizes sending PiM instructions to the PiM inside the memory device over read / write memory transactions, after adjusting the first priority to exceed the second priority.

[0010] In some implementations, the first priority is the PiM instruction routing and execution priority, and the second priority is the memory access transaction routing and execution priority. In some cases, the step of adjusting the first priority to exceed the second priority is performed in a first time period to prioritize PiM instructions over memory access transactions, and the method further includes the step of re-prioritizing memory access transactions over PiM instructions in a second time period by adjusting the second priority to exceed the first priority used to perform compute operations, and the second time period occurs later in time than the first time period. In some implementations, memory transactions are processed on the memory device simultaneously with performing compute operations on the PiM block of the memory device. In some implementations, i) the ML workload is for a large language model ('LLM'); ii) the ML workload corresponds to iterations of LLM; and iii) the performance goal is the end-to-end target completion time for executing iterations of LLM.

[0011] In some cases, the step of determining a potential violation of a performance objective of a PiM processing ML workload may include determining a potential violation in response to determining that the actual time required to execute the ML workload will exceed the end-to-end objective completion time. In some implementations, the step of adjusting a first priority to exceed a second priority may include adjusting a first priority to exceed a second priority based on a Quality of Service ('QoS') policy implemented by a memory controller of the SoC. In some cases, the step of adjusting a first priority to exceed a second priority further includes, based on a QoS policy, placing a PiM instruction between memory transaction requests; routing the PiM instruction placed between memory transaction requests to a memory device; and processing the PiM instruction and the memory transaction request simultaneously at the memory device.

[0012] In some implementations, i) memory read / write transactions are for reading and writing data from the SoC to the memory device, ii) the data read and data write are for data transfer and are distinct from instructions provided to the PiM block within the memory device, and iii) performance goals for executing ML workloads include latency goals for memory read / write transactions and bandwidth or throughput goals for each individual memory read / write transaction.

[0013] Another aspect of the subject matter of the invention described herein may be implemented by a method using a hardware integrated circuit comprising a memory device and a processing-in-memory ('PiM') architecture within the memory device. The method comprises: determining an end-to-end target completion time for executing a PiM operation in the PiM architecture for a set of PiM instructions; and tracking a progress rate (or percentage) of the completed PiM operation relative to an elapsed time rate (or percentage) of the end-to-end target completion time during the execution of the PiM operation in the PiM architecture. The method further comprises: raising the scheduling priority of the remaining PiM operation relative to the priority of the memory transaction for execution in the memory bank of the memory device in response to determining that the progress percentage is less than the elapsed time rate; and executing the memory transaction by using the processing-in-memory architecture to execute the remaining PiM operation and accessing the memory bank of the memory device in response to raising the scheduling priority of the remaining PiM operation.

[0014] Each of these implementations and other implementations may optionally include one or more of the following features. For example, in some implementations, the method further includes the step of determining an elapsed time percentage as a fraction of the elapsed time of completed PiM operations divided by the end-to-end target completion time. In some implementations, the method further includes the step of determining a progress percentage as a fraction of the number of completed operations divided by the sum of the number of completed operations and the number of remaining operations. In some implementations, the method further includes the step of mapping PiM instruction priorities to latency and bandwidth requests. In some cases, the method further includes the step of determining that performance targets for PiM instructions and memory transactions are not met, and the step of increasing the individual operating frequencies of i) the memory controller of the hardware integrated circuit, ii) the memory device, and iii) the processor of the PiM architecture.

[0015] Another aspect of the subject matter of the invention covered herein may be implemented in an integrated circuit comprising: a system-on-chip ('SoC'); a memory device coupled to the SoC; and a processor; and a non-transient machine-readable storage medium for storing instructions executable by the processor to cause the performance of an operation. The operation comprises the steps of: determining a performance goal for executing a machine learning (ML) workload; and executing the ML workload in the integrated circuit by scheduling instructions for compute operations in a processor-in-memory ('PiM') block of the memory device based on a first priority, and determining a potential violation of the performance goal. The method further comprises the steps of: resolving a potential violation by adjusting the first priority to exceed a second priority assigned to a memory read / write transaction provided to the memory device; and satisfying the performance goal of the ML workload in the PiM block and the memory read / write transaction in response to adjusting the first priority to exceed the second priority and adjusting the operating frequency of the memory device and the PiM block.

[0016] Each of these implementations and other implementations may optionally include one or more of the following features. For example, in some implementations, the step of executing an ML workload and meeting performance objectives may include the step of performing any remaining compute operations based on an adjusted first priority that prioritizes sending PiM instructions to a memory device over read / write memory transactions on a memory device, after adjusting the first priority to exceed the second priority. In some cases, the first priority is the PiM instruction routing and execution priority, and the second priority is the memory access transaction routing and execution priority.

[0017] In some implementations, the step of adjusting the first priority to exceed the second priority is performed in a first time period to prioritize PiM instructions over memory access transactions, and the operation further includes the step of re-prioritizing memory access transactions over PiM instructions in a second time period by adjusting the second priority to exceed the first priority used to perform compute operations, wherein the second time period occurs later in time than the first time period. In some implementations, memory transactions to memory are transmitted to the memory device simultaneously with PiM instructions for PiM blocks of the memory device via a shared interface and communication channel.

[0018] In some cases, i) the ML workload is for a large language model ('LLM'); ii) the ML workload corresponds to iterations of the LLM; and iii) the performance goal is the end-to-end target completion time for executing the iterations of the LLM. In some implementations, i) memory read / write transactions are for reading and writing data from the SoC to the memory device; ii) the data read and data write are for data transfer and are distinct from instructions provided to the PiM block within the memory device; and iii) the performance goal for executing the ML workload includes a latency goal for the memory read / write transactions and a bandwidth (or throughput) goal for each individual memory read / write transaction.

[0019] Other embodiments of such and other embodiments of other embodiments include corresponding systems and devices configured to perform the operations of the above method, and corresponding computer programs encoded on a computer storage device. One or more computer systems may be configured to cause the system to perform an operation when operated by software, firmware, hardware, or a combination thereof installed on the system. One or more computer programs may be configured to have instructions that cause the device to perform a corresponding action when executed by a data processing device.

[0020] The subject matter of the invention described herein may be implemented in specific embodiments to realize one or more of the following advantages. The techniques of the present disclosure may be used to address technical problems of inefficiency often present in integrated circuits including SoCs coupled to memory devices having a PiM architecture. Specifically, existing architectures are not known to readily include mechanisms for scheduling PiM-instruction traffic along with normal / routine memory transaction traffic while satisfying performance objectives for workload execution and requirements for PiM operation.

[0021] For example, in existing systems, sources of routine (or regular) memory transaction traffic, such as read-write access requests, may originate from any IP block of the SoC (102), such as an inference accelerator or a graphics / display processor. Memory transactions may be for reading data from a memory location / cell within a DRAM bank of a memory device or for writing data to a memory location / cell. However, the addition of PiM instructions creates new challenges for PiM / SoC architectures. Using the disclosed techniques, the memory controller of the SoC may be configured to support different performance and latency requirements, driving it to prioritize the execution of PiM instructions and operations over memory access transactions.

[0022] Details regarding one or more embodiments of the subject matter of the invention described herein are provided in the accompanying drawings and the following description. Other potential features and advantages of the subject matter of the invention described herein will become apparent from the description, drawings, and claims. Brief explanation of the drawing

[0023] FIG. 1 is a block diagram of an exemplary computing system having at least one SoC. FIG. 2 illustrates an exemplary PiM architecture having corresponding compute elements. Figure 3 illustrates an exemplary timeline sequence of PiM instructions and memory transaction requests. Figure 4 illustrates an exemplary PiM request sequence. Figures 5a and 5b together illustrate examples of changes made to PiM priority based on a comparison of progress rates and elapsed time rates. Figure 6 illustrates an example of merging PiM operations and memory transactions based on changes in execution priority. Figure 7a is an exemplary process for adjusting the priority of PiM operations. FIG. 7b illustrates an exemplary software stack for implementing a policy for PiM command priority adjustment. Figure 8 is an exemplary process for memory operation scheduling for concurrent processing in a memory device. In various drawings, the same reference number and name represent the same element. Specific details for implementing the invention

[0024] FIG. 1 is a block diagram of an exemplary computing system (100) including a system-on-chip (102) ('SoC (102)'). The SoC (102) includes a central processing unit (104) ('CPU (104)'), a memory controller (105), shared memory (106) ('memory (106)'), a resource manager (108), and an IP / circuit block (110). In some implementations, the system (100) may include multiple SoCs, and any description of the SoC (102) will apply equally to each of the multiple SoCs that may be included in the system (100).

[0025] The CPU (104) may be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU (104) generates one or more indicators, such as an app launch indicator or a function call, that are triggered in response to running or launching an application on a user device. For example, the application may be a camera application that uses an imaging sensor to generate image data, or a game application that requires significant memory and graphics processing resources to render graphic content for a game. The CPU (104) also generates one or more application values, such as pixel values ​​or a frame rate. The application values ​​may be associated with a function call, may describe an event that occurs during the execution of the application, or may be both.

[0026] Memory (106) is system memory, shared memory, or both. In the example of FIG. 1, memory (106) is shown outside the circuit block (110). However, memory (106) may include portions of memory that are i) specific to the circuit block (110), ii) outside the circuit block (110), or iii) both. Memory (106) may be random access memory of the SoC (102), such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM. In some implementations, embodiments of memory (106) are configured as shared scratchpad memory that supports parallel access of the memory resources by two or more processors of the circuit block (110). Memory (106) may also include various other types of memory such as high-bandwidth memory (HBM), narrow-width memory (e.g., for storing 8-bit values), wide-width memory (e.g., for storing 16-bit or 32-bit values), etc.

[0027] The resource manager (108) is implemented in hardware and software. Aspects of the resource manager (108) may also be implemented as firmware of the SoC (102) or as firmware of a device of the SoC (102), such as a DRAM memory device or a CPU (104). The resource manager (108) is a processor-in-memory (PiM) resource manager ('PiM resource manager (108)') that includes control logic implemented in hardware, software, or both. For example, the PiM resource manager (108) may include resources such as flip-flops, registers, buffers, etc., implemented in hardware, and control logic (e.g., programmed code) implemented in software.

[0028] A circuit block (110) generally includes individual IP devices such as a processor, a processor core, or a special-purpose processing device. For example, a circuit block (110) may include an image signal processor (ISP) (112), a host (or special-purpose) processing unit (HPU) (114), a digital signal processor (DSP) (116), and a graphics processing unit (GPU) (118). The host device / processor (114) may be a special-purpose neural network processor or a hardware ML accelerator configured to accelerate computations for different types of data processing and generating ML outputs. A circuit block (110) is alternatively referred to as an IP block (110), and an IP block may include one or more proprietary hardware elements. For example, each of the ISP (112), HPU (114), DSP (116), and GPU (118) may be an individual proprietary IP block (or IP device) of a specific entity or device manufacturer.

[0029] One or more aspects of the PiM resource manager (108) may be implemented as a software routine (or module) of the CPU (104) that uses one or more hardware resources of the CPU (104), such as registers, buffers, etc. The CPU (104) may be configured as an instruction and vector data processing engine that processes data obtained from the system memory of the SoC (102), such as memory (106). In some implementations, each processor of the SoC (102) (e.g., ISP (112), DSP (116), HPU (114), GPU (118)) includes multiple cores, and the CPU (104) and / or the PiM resource manager (108) may generate control signaling (124) to manage memory-intensive compute operations and distribute them to memory devices (122) (e.g., DRAM) to minimize the processing load on each core of the processor. Control signaling (124) is routed in the system (100) using an exemplary bus (120) of the SoC (102). Control signaling (124) may include commands, requests, data, instructions, or a combination thereof.

[0030] The PiM resource manager (108) works with the CPU (104), memory controller (105), and storage controller (107) to dynamically control and manage one or more compute-in-memory (CiM) operations. In some implementations, the CiM operations are executed in the SoC (102) to support heterogeneous compute operations between two or more processing units included between the IP block (110), the CPU (104), or both. More specifically, the PiM resource manager (108) is configured to generate control signaling (124) and use one or more individual signal values ​​of the control signaling (124) to manage and boost data access operations in the memory device (122).

[0031] The system (100) includes an exemplary memory device (122). The memory device (122) may include a plurality of memory dies. For example, the memory device (122) may include N memory dies, where N is an integer greater than 1. The memory device (122) may be dynamic random access memory (DRAM) or dual data rate (DDR) synchronous DRAM (SDRAM). The memory device (122) is configured to perform or support various types of PiM operations, CiM operations, and memory-near computing ('MnC') operations. The memory device (122) performs or supports these operations using a plurality of PiM compute elements, which are described below with reference to at least FIG. 2.

[0032] The SoC (102) works in cooperation with the memory device (122) to perform operations across one or more bank groups of the memory device (122). The operations may be for operations or workloads involving one or more processors in the IP block (110). Additionally, the operations may be for heterogeneous operations across multiple processors in the IP block (110), multiple IP blocks (110), or both. In at least one example, the memory device (122) may be outside the SoC (102), whereas in another example, the memory device (122) may be inside the SoC (102).

[0033] In the example of FIG. 1, the system (100) and SoC (102) are integrated circuits of an exemplary user / client device (130), consumer electronic device, or mobile device, each of which may include items such as a smartphone (130a), tablet (130b), laptop (130c), smartwatch, or wearable device (130d). The device (130) may also include other items such as an e-laptop, netbook, smart speaker, or mobile computer. In some implementations, the system (100) and SoC (102) are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0034] FIG. 2 illustrates an exemplary processor-in-memory (PiM) architecture (200) for boosting PiM data access performance based on control signals generated using an SoC (102), a memory device (122), or both. In the example of FIG. 2, the memory device (122) includes a first memory die-1 having a first bank group having a plurality of memory banks, each memory bank comprising one or more memory arrays, and a second memory die-2 having a second bank group having a plurality of memory banks, each memory bank comprising one or more memory arrays. In some implementations, the PiM architecture (200) includes a plurality of bank groups, a plurality of memory dies, or both. For example, a single memory die may include a plurality of bank groups and / or a plurality of bank groups may be distributed across a plurality of memory dies.

[0035] The PiM architecture (200) comprises a plurality of PiM blocks, and each PiM block comprises a plurality of operation elements. For example, a first PiM block of the PiM architecture (200) comprises a mode register (204-1) and a process unit (206-1), while a second different PiM block of the PiM architecture (200) comprises a mode register (204-2) and a process unit (206-2). Each process unit (206-1, 206-2) may comprise a processor, a processor unit, or a processor core, such as a CPU. Each process unit (206-1, 206-2) may also comprise an exemplary operation unit, such as an arithmetic logic unit (ALU) or a multiplication accumulator cell (MAC).

[0036] In some implementations, the PiM architecture (200) is included in the memory device (122) as a plurality of discrete integrated circuits, each integrated circuit being local to a given memory die (e.g., die-1 and die-2) and interacting with or communicating with an array of memory cells on that memory die. For example, the PiM architecture (200) may include computational elements that are replicated and distributed across each memory die within the memory device (122). In some other implementations, the PiM architecture (200) is included in the memory device (122) as a single integrated circuit that interacts with or communicates with each memory die of the memory device (122), including an array of memory cells on each memory die.

[0037] To boost PiM data access performance as described in this specification, a PiM block or process unit within the PiM architecture (200) is located within the memory device (122) but outside a section of the memory device (122) that includes a bank group. The section may be defined as a discrete memory die or in some other way (e.g., a part of a memory die). Regardless of the hardware configuration or layout of the PiM architecture (200), since the PiM block is located sufficiently outside the bank group, the PiM block communicates with the bank group based on specific timing constraints that can be utilized to boost PiM data access performance through data aggregation between bank groups. A data channel / interconnection may be established between the individual memory arrays of the memory banks and the corresponding processor unit and / or mode register of the PiM block (202).

[0038] PiM operations can be managed and executed in a memory device (122) using a processor device that provides functions similar to a central processor, such as a CPU (104). CiM operations and MnC operations include standard arithmetic operations, such as operations typically performed by an ALU or MAC. CiM operations and MnC operations may also include multiplication and addition operations for matrix mathematics, vector operations, linear algebra, and the operation functions of the HPU (114), such as inner product accumulation. In some implementations, each of the PiM operations, CiM operations, and MnC operations is performed by supporting machine learning operations, neural network operations, or both.

[0039] In some implementations, the PiM operation is an extension of the computational function of the HPU (114). For example, the PiM block can generate an accumulator value from a set of weight values / inputs and activation inputs obtained from memory banks of different bank groups within the memory device (122). The accumulator value is generated based on neural network operations performed using the computation array of the PiM block. The computation array may be a matrix multiplication unit having compute cells arranged as a systolic array. The accumulator value may be the inner product of the set of weight values ​​and the activation inputs. That is, for a set of weights, the PiM block multiplies each weight and each activation input and sums the products to form the accumulator value.

[0040] The PiM architecture (200) may include registers or other parts of memory for storing data for individual memory dies or groups of memory banks. For example, the data may be mode / configuration values. The data may also describe errors that occurred during compute operations in the corresponding PiM block of the memory device (122), or both. In some implementations, registers or other parts of memory are used to store configuration information or associated instructions for configuring the PiM block, or the individual memory dies, groups of memory banks, or combinations thereof.

[0041] For example, the mode registers (204-1, 204-2) may be used to control or trigger the selection of a specific mode in the PiM architecture, such as an error capture mode, an interleaved configuration mode, or a multi-batch processing mode. In some implementations, a specific mode is selected based on the bit values ​​of the mode registers (204-1, 204-2). For example, to trigger or select the interleaved configuration mode(s) or the multi-batch processing mode(s), a single bit or a sequence of bits may be defined to be used in the mode registers.

[0042] Using the disclosed techniques, data access operations are optimized by reading memory cells of a bank group at a frequency exceeding other read operations subject to specific delay constraints for executing consecutive reads for a bank of the memory device (122). For example, an internal controller of the PiM block (1) or PiM block (2) may execute consecutive read commands at a frequency based on a clock cycle generated by the memory device (122). The internal controller may operate based on a specific clock frequency, for example, a 200 or 800 MHz clock or a 1000 MHz clock. Other clock frequencies are also within the scope of the present disclosure.

[0043] FIG. 3 illustrates an exemplary timeline sequence (300) of PiM commands and memory transaction requests used to route requests and commands to a memory device (122) via a shared memory path. In the example of FIG. 3, the request sequence (300) includes PiM commands (302) and memory transaction requests (304) from a host device (e.g., HPU (114)) of the SoC (102) over a timeline (306). FIG. 3 indicates that memory transaction requests are memory reads, but it should be noted that memory writes are also supported by the disclosed techniques and are included within the scope of the present disclosure.

[0044] Each group of PiM commands (302) includes activation of multiple banks (308), execution of PiM operations (310), and precharge (312). For example, to accelerate the entire end-to-end execution pipeline, a host (e.g., SoC or HPU (114)) processes compute-intensive workloads and a PiM processes memory-access intensive workloads. In a concurrent use case scenario, multiple IPs (e.g., CPU (104), GPU (118), camera / ISP (112), etc.) may generate memory transaction traffic, such as data read and write, for routing to a memory device (122). The host devices simultaneously generate PiM-related commands for routing to a PiM block (202) of the memory device (122) using the same memory path.

[0045] In the example of FIG. 3, the shared memory path can cause starvation (e.g., bottlenecks and unbalanced processing) in either or both of the PiM instruction traffic and memory access traffic. In some implementations, the priority assigned to PiM instructions is indicated to have a lower priority compared to the priority assigned to regular / routine memory transactions, such as memory transactions other than PiM instructions. Based on this lower priority, the memory controller (105) may be configured to preempt the routing of PiM instructions that trigger the execution of PiM operations in an advantageous way for routing and executing routine memory accesses that require low latency. In some implementations, priority determination, assignment, and priority change (e.g., priority increase or decrease) are executed on the SoC (102) using, for example, the CPU (104), the HPU (114), or both. Priority operations may also be executed by the memory controller (105), independently, or in cooperation with the CPU (104), the HPU (114), or both.

[0046] The techniques of the present disclosure can be used to address technical problems of inefficiency that often exist in integrated circuits including an SoC (102) coupled to a memory device (122) having a PiM architecture. Specifically, existing architectures are not known to readily include a mechanism for scheduling PiM-instruction traffic along with normal / routine memory transaction traffic while meeting performance goals for workload execution and requirements for PiM operation.

[0047] In existing systems, sources of routine memory transaction traffic, such as read-write access requests, for example, may originate from any IP block of the SoC (102), including the CPU (104), GPU (118), ISP (112), HPU (114), and other IP devices such as GSO or DPU. Memory transaction requests may be for reading data from a memory location / cell within a DRAM bank of a memory device (122) or for writing data to a memory location / cell. In some implementations, specific key process indicators (KPIs) associated with executing memory transactions include 1) latency from the read instruction to the client IP block until data is reached, 2) bandwidth or throughput of the accessed data, and 3) DRAM data transfer efficiency (e.g., the actual amount of data transferred divided by the theoretical maximum data transfer rate). However, the addition of PiM instructions creates new challenges for PiM / SoC architectures.

[0048] For example, the memory controller (105) may be configured to support strict latency requirements that induce prioritization of memory access transactions over the execution of PiM commands and operations. In some implementations, the system (100) experiences instances of preemption that cause inefficiency due to an excessive number of activations and precharges across the banks / cells of the memory device (122) rather than actual read / write operations on the banks / cells. For a given memory transaction, the memory device (122) may activate (or open) one or more rows / cells of the memory device (314), perform a read / write operation (316), and then close the rows / cells by precharging the rows / cells (318). In some implementations, activation (308) and precharge (312) are essentially overheads corresponding to a group of one or more PiM operation executions (310) where actual operations occur.

[0049] FIG. 4 illustrates an exemplary PiM request sequence (400). In the example of FIG. 4, at the start of a large language model ('LLM') iteration, a host (e.g., HPU (114)) can determine the end-to-end goal completion time (402) of the iteration. In some implementations, the end-to-end goal completion time (402) can be calculated based on at least the number of tokens per second within the LLM and the total number of PiM actions (310) to be completed.

[0050] During the sequence (400), multiple PiM commands may be transmitted to perform or execute multiple corresponding PiM actions. Multiple PiM commands (302) and PiM actions (310) may be performed to execute an exemplary ML workload. During the execution of the PiM commands (302), the host (e.g., HPU (114)) may track the elapsed time rate (406) and the progress rate compared over time (306). For example, the host or SoC (102) may track the progress rate at a given 'now' time (404).

[0051] In some implementations, the elapsed time ratio (406) is calculated as a fraction of the elapsed time (408) (of the completed operation (410)) divided by the end-to-end target completion time (402). At this time, the remaining operation (412) has not yet been executed. In some implementations, the progress ratio (414) is calculated as a fraction of the number of completed operations (410) divided by the total number of operations or by the sum of the number of completed operations (410) and the number of remaining operations (412). For example, if the operation (410) includes 100 operation tasks, the progress ratio (414) can be calculated as N / 100, where N is the number of completed operations (410).

[0052] FIGS. 5a and 5b together illustrate an example of a change made to the PiM priority based on a comparison of the progress rate (414) and the elapsed time rate (406). For example, this comparison may be made with reference to the PiM command to be executed with reference to FIG. 4. Referring to FIG. 5a, if the progress rate (414) is lower than the elapsed time rate (406), the priority assigned to the PiM command is increased. This may occur, for example, when there is a possibility of missing the target completion time. Increasing the priority assigned to the PiM command in this way will give the PiM command priority over the memory transaction request (304). In some implementations, the time rate and the progress rate may each be computed, determined, or otherwise characterized as a percentage value or a ratio value.

[0053] In some implementations, the SoC (102) generates different priority parameters based on prioritization logic (or algorithm) executed using the CPU (104), the memory controller (105), or both. In some cases, the SoC (102) generates individual priorities and corresponding priority parameters for different workloads and memory transactions based on a prioritization method executed using one or more of the CPU (104), the resource manager (108), the host processor of the SoC (102), the memory controller (105), or the PiM blocks of the PiM architecture within the memory device (122).

[0054] For example, the SoC (102) may execute different workloads, such as an image processing workload for the ISP (112) or a portion of an LLM workload executed using the HPU (114). The SoC (102) (e.g., CPU (104)) may set individual priorities for each workload based, for example, on the corresponding performance goals or requirements for each workload. For example, the prioritization logic may receive performance goals corresponding to the ISP workload (e.g., frames per second (fps) sensitivity) and the LLM workload (e.g., latency sensitivity), respectively, and use the respective goal values ​​to establish individual priorities for each workload according to the corresponding performance goals. For example, the performance goal for the image processing workload may specify a threshold fps requirement, whereas the performance goal for the LLM workload may be latency sensitive based on a minimum requirement for computing word tokens to present a generated response to the user.

[0055] In some implementations, priority logic is used by the SoC (102) to establish individual priorities for different memory transactions mediated by the memory controller (105). For example, different memory transactions may be associated with one of two corresponding workloads and may be required to execute said corresponding workload according to associated performance goals. In some implementations, the SoC (102) may generate priority parameters based on the priority logic. For example, the memory controller (105) may generate 'HIGH' or 'LOW' priority parameters using a 1-bit flag or generate more granular priority parameters using an N-bit parameter, where N is an integer greater than 1. In some implementations, priority parameters indicating the corresponding priority of a PiM instruction are passed to the PiM block and stored as configuration settings of the control (or mode) register of the PiM block. For example, priority parameters are passed to the PiM block by the memory controller (105).

[0056] Referring to FIG. 5b, if the progress rate (414) is higher than the elapsed time rate (406), the priority assigned to the PiM command is reduced. This may occur, for example, when there is a possibility that the PiM command will be completed earlier than the target completion time. Reducing the priority assigned to the PiM command will give priority to the routine memory read request (304) over the PiM command. By doing so, the memory read request (304) can be performed without negatively affecting the possibility that the PiM command will still be completed before the end of the target completion time. That is, the memory transaction, for example, the memory read request (304), may be reassigned to priority over the PiM command by adjusting the priority assigned to the memory transaction to exceed or be higher than the priority assigned to the PiM command.

[0057] In some cases, adjusting the first priority to exceed the second priority occurs in the first time period to prioritize PiM instructions over memory access transactions. Then, the SoC (102) may re-prioritize memory access transactions over PiM instructions in the second time period by adjusting the second priority to exceed the first priority used to perform compute operations. The second time period occurs later in time than the first time period. For example, the memory controller (105) may decide to re-prioritize memory access transactions over PiM instructions by adjusting the second priority to exceed the first priority in response to determining that a potential violation has been resolved. The potential violation may be resolved (e.g., mitigated) based on the previous change in which the first priority was adjusted to exceed the second priority. For example, the potential violation may be resolved based on an updated calculation indicating that the execution progress of the PiM instruction for an in-memory compute iteration is higher than the elapsed time ratio required to execute the instruction and compute operation according to the performance objective.

[0058] In some implementations, changes to priority based on a comparison of the progress rate (414) and the elapsed time rate (406) may be limited. For example, if the progress rate (414) is within a threshold (e.g., 5%) of the elapsed time rate (406), it may be decided not to change the priority.

[0059] FIG. 6 illustrates an example of request merging increase resulting from changing the PiM command priority based on a comparison of the progress rate (414) and the elapsed time rate (406).

[0060] The disclosed techniques provide a mechanism for efficiently scheduling PiM-instruction traffic along with memory access transaction traffic. In some implementations, memory access transaction traffic is alternatively referred to as regular memory transactions or regular memory transaction traffic. Conventional approaches in existing systems include QoS schemes for routine read / write memory access traffic only. KPIs for routine (or regular) memory transaction traffic generally include latency and bandwidth requirements. However, PiM instruction traffic from a host device routed to a PiM block of a memory device (122) will have different performance requirements that may vary based on execution conditions and use cases. Accordingly, techniques for a memory controller (105) are disclosed that provide a mechanism for scheduling and prioritizing PiM instruction executions to meet ML workload / PiM / HPU performance requirements.

[0061] For example, as described with reference to FIGS. 5a and 5b, if the priority assigned to a PiM instruction is higher than that of other memory transaction requests / accesses, the PiM operation (or corresponding PiM instruction) will be scheduled and executed sequentially. This improves performance and efficiency when executing PiM operations. In some implementations, this improvement is at least partially due to a reduction in overall overhead. This provides for an increase in the merging of PiM operations and memory accesses. That is, rather than interleaving different types of operations, it may be more efficient to perform a larger number of similar operations (e.g., PiM operations or memory accesses) in blocks.

[0062] In some implementations, PiM command priorities can be mapped to latency and bandwidth requests in a QoS manner that may include exemplary settings in the QoS configuration. For example, during the execution of a PiM command, a host (e.g., HPU (114)) can track the progress rate and compare the progress rate to the elapsed time rate. If the progress rate is lower than the elapsed time rate, the priority assigned to the PiM command may be increased. If the progress rate is higher than the elapsed time rate, the priority of the PiM command is decreased.

[0063] FIG. 7a is an exemplary workflow / process (700) for adjusting the priority of PiM requests, for example, for controlling memory traffic scheduling in PiM. In step 702, at the start of an LLM iteration, a host (e.g., HPU (114)) determines the end-to-end target completion time of the iteration. In step 704, during the execution of a PiM command, the host (e.g., HPU (114)) can track the progress rate and compare the progress rate with the elapsed time rate.

[0064] In step 706, if the progress rate is lower than the elapsed time rate, the priority assigned to the PiM instruction is increased. That is, the priority assigned to the PiM instruction is increased to a higher level than the priority assigned to the regular memory transaction.

[0065] For example, a PiM command may initially be assigned a first priority (P3) used by the memory controller (105) to prioritize and / or mediate the execution of different memory transaction requests from devices of the SoC (102). Then, the system (100) may increase the priority assigned to the PiM command to ensure that the execution (and / or routing) of the PiM command meets performance objectives. For example, to increase the priority, the system (100) may generate and assign a second different priority (P5) to the PiM command, wherein the second priority (P5) is a higher priority than the first priority (P3). In some implementations, the system (100) may adjust the priority assigned to routing the PiM command to the memory device (122), adjust the priority assigned to executing the PiM command on the memory device (122), or perform both.

[0066] In step 708, if the progress rate is higher than the elapsed time rate, the priority of the PiM command is reduced. That is, the priority assigned to the PiM command is reduced to a lower level compared to the priority assigned to the regular memory transaction. For example, the PiM command may initially be assigned a first priority (P5) used by the memory controller (105) to prioritize and / or mediate the execution of different memory transaction requests from devices of the SoC (102). Then, the system (100) may reduce the priority assigned to the PiM command to ensure that the execution (and / or routing) of the PiM command meets performance objectives. For example, to reduce the priority, the system (100) may generate a second different priority (P3) and assign it to the PiM command, wherein the second priority (P3) is a lower priority than the first priority (P5).

[0067] At step 710, PiM instruction priority is mapped to latency and bandwidth requests in the memory controller (105). At step 712, if the system (100) determines that it is not possible to meet (or exceed) the performance target requests of PiM instructions and regular memory access transactions / traffic, the operating frequency of the memory controller (105) and the memory device (122) is increased. For example, the memory controller (105) may generate and / or transmit a control signal to the memory device (122) to boost the operating frequency in the memory device (122). The mode or control register of the PiM block may store and / or generate a control signal used to increase the operating frequency of the PiM architecture to enable the system (100) to meet the performance target of the ML workload running in the PiM block. The workload may be executed concurrently with the execution of memory read / write transactions in the memory device (122).

[0068] The operating frequency of the PiM architecture (200) may be increased or boosted in response to adjusting the first priority to exceed the second priority. For example, adjusting one priority to exceed another priority may include and / or trigger adjusting the operating frequency of the PiM block as well as the operating frequency of other sections of the memory device (122). In some implementations, the change in the operating frequency of the memory device (122) may be performed conditionally, regardless of the priority change. For example, the operating frequency may be changed or adjusted to ensure that the current set of PiM instructions is routed and / or executed to meet or exceed its performance goals according to the existing priority.

[0069] FIG. 7b illustrates an exemplary diagram of a host device software stack (720) implementing an exemplary policy (722) for PiM command priority adjustment. In some implementations, one or more of steps (1) through (4) of the policy (722) correspond to one or more steps of the process (700). The software stack (720) includes an application delegate (724), a host runtime block (726), a host device kernel driver (728), and host device firmware (730). The application delegate (724) is used to implement item (1) of the policy (722) and includes application software (e.g., TensorFlow-lite) and an AI / ML neural network model that may be use case-specific.

[0070] The host runtime block (726) and the host device kernel driver (728) are used to implement items (2, 3a, 3b) of the policy (722). In some implementations, the host runtime block (726) is a runtime application programming interface (API) that provides an abstraction for an application to configure a host processing unit driver, register an ML model, and execute inference using the ML model. In this regard, the host device kernel driver (728) may be a programming interface that allows communication between the device hardware and a specific higher-level software application. The host device firmware (730) is used to implement item (4) of the policy (722). In some implementations, the host device firmware (730) is low-level software that directly controls hardware aspects of the host device, including setting specific registers and / or register values.

[0071] FIG. 8 is an exemplary process (800) for memory access scheduling in an integrated computing system, including memory transaction / operation scheduling for concurrent processing in a DRAM device such as a memory device (122). The process (800) is implemented or executed in the system (100) using at least the SoC (102) and memory device (122) described above. Accordingly, the description of the process (800) will refer to the computing resources mentioned above of the system (100). In some examples, the steps or actions of the process (800) may be enabled by programmed software instructions, firmware instructions, or both. Instructions of each type may be stored in a non-transient machine-readable storage device and may be executed by one or more processors or other resources described herein.

[0072] Referring again to process (800), the system (100) determines a performance goal for executing a machine learning (ML) workload (802). For example, the ML workload may be for a large language model ('LLM') executed by a host device of the SoC (102), such as an HPU (114). In some implementations, the ML workload corresponds to iterations of the LLM, and the performance goal is the end-to-end target completion time for executing the iterations of the LLM.

[0073] To execute an ML workload in an integrated circuit system, the system (100) performs compute operations in a PiM block (202) of a memory device (122) based on a first priority (804). For example, the PiM block (202) may perform operations to generate an accumulator value from an activation input and a set of weight values ​​obtained from memory banks of different bank groups within the memory device (122). In some implementations, the accumulator value is generated based on a neural network operation performed using an array of operations in the PiM block (202).

[0074] The system (100) determines a potential violation of a performance goal (806). For example, the system (100) may determine a potential violation in response to determining that the actual time required to execute an ML workload will exceed the end-to-end goal completion time. The system (100) resolves the potential violation by adjusting the first priority to exceed the second priority assigned to memory transactions processed in the memory device (122) (808).

[0075] The system executes an ML workload and meets performance objectives in response to adjusting the first priority to exceed the second priority (810). For example, after adjusting the first priority to exceed the second priority, the system (100) performs any remaining compute operations based on the adjusted first priority. Adjusting the first priority to exceed the second priority causes the memory controller (105) to prioritize the execution of PiM instructions in the memory device (122) over the execution of memory transactions in the memory device (122). Prioritizing the execution of PiM instructions over one or more other memory device operations enables the ML workload to be executed while also meeting performance objectives.

[0076] The first priority may be a PiM command routing and execution priority, while the second priority may be a memory access transaction routing and execution priority. In some implementations, the system (100) initially prioritizes executing memory transactions on the memory device (122) over executing PiM commands on the memory device (122). The system (100) may set this initial priority by configuring the second priority to exceed the first priority so that the SoC (102) prioritizes memory transaction requests from the IP device (108) requesting read / write access to the memory bank of the memory device (122).

[0077] Next, the system (100) may prioritize the execution of PiM instructions in the memory device over the execution of memory transactions in the memory device (122) by adjusting the first priority to exceed the second priority. For example, as shown above, the system (100) may configure this priority change in response to a determination that the actual time required to execute the ML workload will violate or deviate from the performance goal. In some implementations, the execution of the ML workload may be very fast (or may be faster) so that prioritizing the execution of PiM instructions over the execution of memory transactions is not required (or is no longer required) to meet the performance goal.

[0078] In such cases, the system (100) may re-prioritize memory transactions over executing PiM instructions by adjusting the second priority to exceed the first priority used to perform compute operations. In some implementations, the system (100) makes adjustments to the first priority or the second priority based on a quality of service ('QoS') policy implemented by the memory controller (105) of the SoC (102).

[0079] In concurrent use case scenarios where multiple IPs (e.g., CPU (104), GPU (118), camera / ISP (112), etc.) generate memory transaction traffic, such as data read and write, for routing to a memory device (122), a host device (e.g., HPU (114)) may simultaneously generate PiM-related commands for routing to a PiM block (202) of the memory device (122) using the same memory path. Thus, PiM commands and routine memory read / write transactions may share a common set of interface communication channels existing between the SoC (102) and the memory device (122). The system (100) may be configured to distribute and / or interleave the timing of routing PiM commands and routine memory access transactions (e.g., read / write traffic) to the memory device (122). For example, the memory controller (105) can distribute and / or interleave routing timing so that PiM commands and memory access operations can be processed simultaneously on the memory device (122) for one or more tasks of an ML workload running on the memory device (122).

[0080] The objects of the invention and embodiments of functional operation described herein may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including structures disclosed herein and structural equivalents thereof, or in a combination of one or more of these. The embodiments of the objects of the invention described herein may be implemented as one or more computer programs, namely, as one or more modules of computer program instructions encoded on a tangible non-transient program carrier for execution by a data processing device or for controlling the operation of a data processing device.

[0081] Alternatively or additionally, program instructions may be encoded in artificially generated radio signals, for example, mechanically generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a receiver device suitable for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0082] The term "computing system" encompasses all types of devices, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may be or may include special-purpose logic circuits, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A device may also include, in addition to the hardware, code that creates an execution environment for the corresponding computer program, for example, processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0083] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including a compiled or interpreted language, or a declarative or procedural language, and may be placed in any form as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0084] A computer program may correspond to a file within a file system, but it is not necessarily required to be so. The program may be stored in a part of a file containing other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple coordinated files, for example, files storing one or more modules, subprograms, or parts of code. The computer program may be deployed to be executed on a single computer or a single site, or on multiple computers distributed across multiple sites and interconnected by a communication network.

[0085] The processes and logic flows described herein may be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by special purpose logic circuits, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a general purpose graphics processing unit (GPGPU), and the device may also be implemented as these.

[0086] A computer suitable for executing computer programs may be based, for example, on a general-purpose or special-purpose microprocessor, or both, or on any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory or random access memory, or both. Some elements of the computer are the central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, the computer will also include one or more mass storage devices for storing data, for example, magnetic, magneto-optical disks, or optical disks, or will be operably coupled to receive data from them, transmit data to them, or perform both. However, the computer does not need to have such devices. Additionally, the computer may be embedded in other devices, for example, to name just a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive.

[0087] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or integrated therein by special-purpose logic circuits.

[0088] To provide interaction with a user, embodiments of the subject matter of the invention described herein may be implemented in a computer having a display device for displaying information to a user, e.g., a liquid crystal display (LCD) monitor, and a keyboard and pointing device, e.g., a mouse or a trackball, for which a user can provide input to the computer. Other types of devices may also be used to provide interaction with a user; for example, feedback provided to a user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from a user may be received in any form, including sound, voice, or tactile input. Additionally, the computer may interact with the user by sending documents to a device used by the user and receiving documents from thereon; for example, it may interact by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0089] An embodiment of the subject of the invention described herein may be implemented in a computing system comprising a backend component, e.g., a data server, or a computing system comprising a middleware component, e.g., an application server, or a frontend component, e.g., a graphical user interface, or a computing system comprising a client computer equipped with a web browser that allows a user to interact with an implementation of the subject of the invention described herein, or in a combination of one or more of such backend, middleware, or frontend components. Components of the system may be interconnected through any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks ('LAN') and wide area networks ('WAN'), e.g., the Internet.

[0090] A computing system may include clients and servers. Clients and servers are typically separated from each other and generally interact through a communication network. The relationship between a client and a server is generated by computer programs that operate on individual computers and have a client-server relationship with each other.

[0091] Although this specification contains many specific implementation details, they should not be interpreted as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in a specific combination and may even be initially claimed as such, one or more features from the claimed combination may be omitted from the combination in some cases, and the claimed combination may relate to a sub-combination or a variation of a sub-combination.

[0092] Similarly, although operations are depicted in a specific order in the drawings, this should not be understood as requiring that such operations be performed in the specific order depicted or in a sequential order, or that all illustrated operations be performed, in order to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the aforementioned embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0093] Specific embodiments of the subject matter of the invention have been described. Other embodiments are included in the following claims. For example, the actions described in the claims may be performed in different orders and still achieve desirable results. As an example, the processes illustrated in the accompanying drawings do not necessarily require the specific order or sequential order illustrated to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

Claim 1 A method implemented using an integrated circuit comprising a system-on-chip ('SoC') and a memory device coupled to the SoC, comprising: determining a performance objective for executing a machine learning (ML) workload; executing the ML workload on the integrated circuit by scheduling instructions for compute operations in a processor-in-memory ('PiM') block of the memory device based on a first priority; determining a potential violation of the performance objective; resolving the potential violation by adjusting the first priority to exceed a second priority assigned to a memory read / write transaction provided to the memory device; and satisfying the performance objective of the ML workload and the memory read / write transaction in the PiM block in response to adjusting the first priority to exceed the second priority and adjusting the operating frequency of the memory device and the PiM block. Claim 2 A method according to claim 1, wherein i) the command and the memory read / write transaction are processed simultaneously for at least one task of the ML workload; and ii) the command and the memory read / write transaction share an interface communication channel between the SoC and the memory device. Claim 3 A method according to claim 2, wherein the step of executing the ML workload and satisfying the performance objective comprises the step of performing any remaining compute operation based on an adjusted first priority that prioritizes sending PiM commands to the PiM inside the memory device over sending read / write memory transactions after adjusting the first priority to exceed the second priority. Claim 4 A method according to any one of claims 1 to 3, wherein the first priority is a PiM instruction routing and execution priority; and the second priority is a memory access transaction routing and execution priority. Claim 5 In claim 4, the step of adjusting the first priority to exceed the second priority occurs in a first time period to prioritize PiM instructions over memory access transactions, and the method includes the step of re-prioritizing memory access transactions over PiM instructions in a second time period by adjusting the second priority to exceed the first priority used to perform the compute operation, wherein the second time period occurs later in time than the first time period. Claim 6 A method according to any one of claims 1 to 5, wherein the memory transaction is processed in the memory device simultaneously with performing the compute operation in the PiM block of the memory device. Claim 7 A method according to any one of claims 1 through 6, wherein i) the ML workload is for a large-scale language model ('LLM'); ii) the ML workload corresponds to iterations of the LLM; and iii) the performance goal is an end-to-end target completion time for executing iterations of the LLM. Claim 8 In claim 7, the step of determining a potential violation of the performance objective of the PiM processing ML workload comprises determining a potential violation in response to determining that the actual time required to execute the ML workload will exceed the end-to-end objective completion time. Claim 9 A method according to any one of claims 1 to 8, wherein the step of adjusting the first priority to exceed the second priority includes the step of adjusting the first priority to exceed the second priority based on a quality of service ('QoS') policy implemented by the memory controller of the SoC. Claim 10 A method according to claim 9, further comprising: a step of placing a PiM command between memory transaction requests based on the above QoS policy; a step of routing the PiM command placed between memory transaction requests to a memory device; and a step of simultaneously processing the PiM command and the memory transaction request in the memory device. Claim 11 A method according to any one of claims 1 to 10, wherein i) the memory read / write transaction is for reading and writing data from the SoC to the memory device; ii) the data read and data write is for data transfer and is distinct from instructions provided to the PiM block within the memory device; and iii) the performance goal for executing the ML workload includes a latency goal for the memory read / write transaction and a bandwidth or throughput goal for each individual memory read / write transaction. Claim 12 A method implemented using a hardware integrated circuit comprising a memory device and a processing-in-memory ('PiM') architecture within the memory device, comprising: determining an end-to-end target completion time for executing a PiM operation in the PiM architecture for a set of PiM instructions; tracking a progress rate of the completed PiM operation relative to a ratio of elapsed time of the end-to-end target completion time during the execution of the PiM operation in the PiM architecture; increasing the scheduling priority of the remaining PiM operation relative to the priority of the memory transaction for execution in the memory bank of the memory device in response to determining that the progress rate is lower than the ratio of elapsed time; and executing the memory transaction by executing the remaining PiM operation using the processing-in-memory architecture and accessing the memory bank of the memory device in response to increasing the scheduling priority of the remaining PiM operation. Claim 13 A method according to claim 12, further comprising the step of determining the elapsed time ratio as a fraction of the elapsed time of the completed PiM operation divided by the end-to-end target completion time. Claim 14 A method according to claim 13, further comprising the step of determining the progress ratio as a fraction obtained by dividing the number of completed operations by the sum of the number of completed operations and the number of remaining operations. Claim 15 A method comprising, in any one of claims 12 to 14, further a step of mapping PiM command priority to latency and bandwidth requests. Claim 16 A method comprising, in any one of claims 12 to 15, a step of determining that the performance objectives of the PiM instruction and memory transaction are not satisfied; and further comprising the step of increasing the individual operating frequencies of the memory controller of the hardware integrated circuit, the memory device, and the processor of the PiM architecture. Claim 17 An integrated circuit comprising: a system-on-chip ('SoC'); a memory device coupled to the SoC; and a processor and a non-transient machine-readable storage medium—the non-transient machine-readable storage medium stores instructions executable by the processor to cause the execution of an operation—the operation comprises: determining a performance goal for executing a machine learning (ML) workload; executing the ML workload in the integrated circuit by scheduling instructions for a compute operation in a processor-in-memory ('PiM') block of the memory device based on a first priority; determining a potential violation of the performance goal; resolving the potential violation by adjusting the first priority to exceed a second priority assigned to a memory read / write transaction provided to the memory device; and satisfying the performance goal of the ML workload and the memory read / write transaction in the PiM block in response to adjusting the first priority to exceed the second priority and adjusting the operating frequency of the memory device and the PiM block. Claim 18 An integrated circuit according to claim 17, wherein the step of executing the ML workload and satisfying the performance objective comprises the step of performing any remaining compute operation based on an adjusted first priority that prioritizes sending PiM commands to the memory device over memory read / write transactions in the memory device after adjusting the first priority to exceed the second priority. Claim 19 An integrated circuit, wherein, in paragraph 17 or 18, the first priority is a PiM instruction routing and execution priority; and the second priority is a memory access transaction routing and execution priority. Claim 20 In claim 19, the step of adjusting the first priority to exceed the second priority occurs in a first time period to prioritize PiM instructions over memory access transactions, and the operation includes the step of re-prioritizing memory access transactions over PiM instructions in a second time period by adjusting the second priority to exceed the first priority used to perform the compute operation, wherein the second time period occurs later in time than the first time period. Claim 21 An integrated circuit according to any one of claims 17 to 20, wherein the memory transaction to the memory is transmitted to the memory device simultaneously with the PiM command for the PiM block of the memory device through a shared interface and a communication channel. Claim 22 An integrated circuit according to any one of claims 17 through 21, wherein i) the ML workload is for a large-scale language model ('LLM'); ii) the ML workload corresponds to iterations of the LLM; and iii) the performance objective is an end-to-end target completion time for executing iterations of the LLM. Claim 23 An integrated circuit according to any one of claims 17 through 22, wherein i) the memory read / write transaction is for reading and writing data from the SoC to the memory device; ii) the data read and data write is for data transfer and is distinct from instructions provided to the PiM block within the memory device; and iii) the performance goal for executing the ML workload includes a latency goal for the memory read / write transaction and a bandwidth (or throughput) goal for each individual memory read / write transaction.

Images