Collaborative DVFS control for a processing-in-memory architecture of a heterogeneous computing system

Abstract

Methods, systems, and apparatus, including computational instructions / programs encoded on non-transitory computer-readable media, are disclosed to implement collaborative controls for dynamic voltage & frequency scaling ("DVFS") using an integrated circuit comprising a System-on-Chip ("SoC") and a memory device coupled to the SoC. The system identifies user-experience criteria for generating an output of a machine-learning ("ML") model implemented at the integrated circuit and determines a performance target required to satisfy the user-experience criteria. The system computes a first target operating point of a processing-in-memory ("PiM") block inside the memory device and a second target operating point of the host core. The system performs computations at the PiM block using the target operating point established based on the control signals. The first and second target operating points achieve the best energy efficiency, while the PiM block and the host core collaboratively satisfy the user-experience criteria.

Description

COLLABORATIVE DVFS CONTROL FOR A PROCESSING-IN-MEMORY ARCHITECTURE OF A HETEROGENEOUS COMPUTING SYSTEMBACKGROUND

[0001] This specification generally relates to memory devices used to execute computations.

[0002] Modem computing systems often incorporate a wide variety of compute processing units that each offer different computing capabilities and trade-offs. Efficient execution of a given compute job often involves parsing computations into meaningful subtasks or workloads that are mapped to available processor cores of a computing system. The computations may be parsed and mapped based on suitability criteria, such as processor capability', performance, and power. Generally, this overall process of allocating portions of a computation to appropriate processor resources is referred to as a heterogeneous computation.

[0003] At least one processor core of the computing system can be an Intellectual Property block (“IP block”) that executes a respective portion of a computational operation for different multimedia workloads. Example use cases can involve processing image or speech data captured respectively by a camera or microphone on the mobile device as well as performing computations for generative artificial intelligence (“GenAI”) applications. The system-on-chip can use a heterogeneous compute operation to process input samples derived from image data, speech data, a text corpus, or a combination of these. An example step in the heterogeneous compute operation can include processing data associated with the input samples using a memory device that provides in-memory processing or computing capabilities.SUMMARY

[0004] This specification describes techniques for implementing collaborative controls for dynamic voltage & frequency scaling (“DVFS”) at an integrated system that includes data processing resources in a memory device coupled to a system-on-chip (“SoC”). The memory' device can be a dynamic random-access memory (DRAM) device that includes a Processing- in-Memory (PiM) architecture configured to perform memory' -bounded and compute- bounded computations for executing an inference workload or task in the memory device.

[0005] The PiM architecture defines one or more PiM blocks of the memory device and each PiM block includes computing resources / elements, such as a processor unit, mode registers, and one or more computational units, e.g.. arithmetic logic units (ALUs) or relatedaddition and multiplication circuitry. The PiM blocks are used to execute computations for an example workload that originates at the SoC. The computations can be segmented into respective portions that are allocated between the SoC and the memory device that includes the PiM blocks. The workload can be for a machine-learning (“ML”) model, such as a large language model (“LLM”) used to implement a GenAI application.

[0006] The disclosed techniques for collaborative DVFS controls are used to implement (or run) the ML model on a special-purpose integrated circuit of the system and perform computations for an ML workload with improved efficiency over prior approaches. The ML model executes the workload by performing computations to generate a corresponding model output from an input sample, consistent with certain user-experience criteria. In a GenAI example, generating the model output can include displaying a sequence of text that is responsive to an input sample such as a user query. The system identifies the user-experience criteria for generating the model output, where the criteria can be a minimum number of words or output tokens to be displayed per minute (e.g., 200).

[0007] The system uses the collaborative DVFS controls to: i) determine a performance target required to satisfy the user-experience criteria; ii) compute a target operating point of the memory device required to satisfy the performance target; and ii) generate control signals that establish the target operating point at a PiM block of the memory' device. The system performs computations at the PiM block using the target operating point established based on the control signals. The collaborative DVFS controls can establish an efficient system-level DVFS setting that includes a minimum operating frequency and power output of the PiM block. The DVFS setting provides minimum operating points for efficiently executing the computations to generate the model output, while still satisfying the performance target required to meet the user-experience criteria.

[0008] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0009] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0010] Prior / conventional approaches for integrated system-level controls are limited to running a processing block of a memory device at only one frequency. In contrast to these approaches, the disclosed techniques provide a hardware / software scheme for dynamically and collaboratively controlling operating frequency and voltage (DVFS settings) of a PiM block in an integrated random-access memory device that couples to an SoC. The hardware / software techniques can be leveraged to determine a balance, or establish tradeoffs, between the overall system-level performance & power consumption of a host device and PiM block. In determining the balance, control logic of the techniques evaluate and dynamically tune operating points to establish optimal performance & power ratios that allow for efficient processing of ML workloads while also satisfying user-experience criteria.

[0011] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Fig. 1 is a block diagram of an example computing system with at least one SoC.

[0013] Fig. 2 shows an example PiM architecture with corresponding compute elements.

[0014] Fig. 3 illustrates an example implementation of frequency voting for collaborative DVFS control at the system of Fig. 1.

[0015] Fig. 4 illustrates example graphical data indicating system-level operating points for a host device and a corresponding PiM block.

[0016] Fig. 5 shows examples of user-experience criteria and corresponding performance targets.

[0017] Fig. 6 is a first example process for collaborative DVFS control at the system of Fig. 1.

[0018] Fig. 7 is a second example process for collaborative DVFS control at the sy stem of Fig. 1.

[0019] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0020] Fig. 1 is a block diagram of an example computing system 100 that includes a system-on-chip 102 (“SoC 102”). The SoC 102 includes a central processing unit 104 (“CPU104”), a memory controller 105, a shared memory 106 (“memory 106”), a resource manager 108, and an IP / circuit block 110. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at system 100.

[0021] The CPU 104 can be a general-purpose CPU (e.g., a single or multi-core CPU). The CPU 104 generates one or more indicators, such as an app-launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. The CPU 104 also generates one or more application values, such as pixel values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0022] The memory 106 is a system memory7, shared memory , or both. In the example of Fig. 1, memory 106 is depicted external to circuit block 110. However, memory 106 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory7106 can be random access memory of the SoC 102, such as static random access memory7(SRAM), dynamic random access memory7(DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0023] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit 110. Memory7106 can also include various other types of memory', such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), or a combination of these.

[0024] The resource manager 108 is implemented in hardware and software. Aspects of the resource manager 108 can be also implemented as firmware of the SoC 102 or firmware of a device of the SoC 102, such as a DRAM memory device or the CPU 104. The resource manager 108 is a processor-in-memory (PiM) resource manager (“PiM resource manager 108”) that includes control logic implemented in hardware, software, or both. For example, the PiM resource manager 108 can include resources such as flip-flops, registers, buffers, cache units, or a combination of these, which may be implemented in hardware, along with related control logic (e.g., programmed code), which may be implemented in software as well as hardware.

[0025] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include an image signal processor (ISP) 112, a host processing unit (HPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, HPU 114. DSP 116, and GPU 118 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0026] The HPU 114 can be a special-purpose processor, such as machine-learning hardware accelerator, neural processor unit, neural network processor, or an applicationspecific processor. In some implementations, the HPU 114 is a neural network tensor processor that includes an integrated circuit architecture that is optimized for power-efficient execution of machine-learning computations using tensor constructs such as multidimensional matrices.

[0027] One or more aspects of the PiM resource manager 108 can be implemented as a software routine (or module) of the CPU 104. which uses one or more hardware resources of the CPU 104, such as registers, buffers, etc. The CPU 104 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 102, such as memory 106. In some implementations, each processor (e.g.. ISP 1 12, DSP 116. HPU 114, GPU 118) of the SoC 102 includes multiple cores. The CPU 104 and / or the PiM resource manager 108 can generate control signaling 124 to manage and distribute memory intensive compute operations to a memory device 122 (e.g., DRAM) to minimize the processing load at each core of the processors. The control signaling 124 is routed at system 100 using an example bus 120 of the SoC 102. The control signaling 124 can include commands, requests, data, instructions, or combination of these. In some implementations, the SoC bus 120 includes or corresponds to the example system fabrics described below with reference to Fig. 3.

[0028] The PiM resource manager 108 cooperates with the CPU 104. memory controller 105 and storage controller 107 to dynamically control and manage one or more compute-inmemory (CIM) operations. In some implementations, the CIM operations are executed at the SoC 102 in support of a heterogeneous compute operation between two or more processing units that are included among the IP block 110, the CPU 104, or both. More specifically, the PiM resource manager 108 is configured to generate control signaling 124 and use one ormore discrete signal values of the control signaling 124 to manage, configure, and / or boost data access operations at the memory device 122.

[0029] In general, techniques for collaborative DVFS controls implemented at system 100 can include generating data and control signaling at the SoC 102, which are used to communicate with the PiM architecture and memory device 122 by way of a memory' controller 105 of the SoC 105. For example, the data and control signaling 124 generated at the SoC 102 are passed to. and processed by, compute elements of the PiM blocks 202 within a given PiM architecture 200 to execute the computations for an example ML inference workload. In some implementations, the data / control signaling 124 are processed at the PiM architecture 200 to trigger execution of certain data processing and computing operations, including DVFS control operations, using compute intervals that represent different pipeline stages of the PiM blocks 202(1) and 202(2). Control signals can also be generated locally at the memory device 122, which may be external to, or separate from, the SoC 102.

[0030] The system 100 includes an example memory device 122. The memory device 122 can include multiple memory dies. For example, the memory’ device 122 can include N memory die. where N is an integer greater than 1. The memory device 122 can be a dynamic random-access memory (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory' device 122 is configured to perform or support various ty pes of PiM operations, CiM operations, and memory-near-computing operations (“MnC operations’7). The memory device 122 performs or supports these operations using its multiple PiM compute elements, which are described below yvith reference to Fig. 2.

[0031] The SoC 102 cooperates yvith the memory device 122 to perform computations across one or more bank groups of the memory device 122. The computations can be for operations or workloads that involve one or more of the processors at IP block 110. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In some implementations, the heterogeneous operation includes (or spans) a host device, e.g., HPU 114 and the memory device 122 cooperating to execute an example ML yvorkload. In at least one example the memory device 122 may be external to the SoC 102, whereas in another example the memory device 122 may be internal to the SoC 102. The terms “host device” and “host core” may be used interchangeably in the present disclosure. In some implementations, the SoC 102 is integrated (or co-located) yvith the memory' device 122, for example, as distinct circuit die(s) that are co-located in a single integrated circuit package.

[0032] In the example of Fig. 1, system 100 and the SoC 102 is an integrated circuit of an example user / client device 130, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 130a, tablet 130b, laptop 130c, smartwatch or wearable device 130d. The devices 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0033] Fig. 2 shows an example processor-in-memory (PiM) architecture 200 for boosting PiM data access performance based on control signals generated using the SoC 102, the memory device 122, or both. In the example of Fig. 2, the memory device 122 includes a first memory die-1 with a first bank group that has multiple memory banks, where each memory bank includes one or more memory arrays and a second memory' die-2 with a second bank group that has multiple memory banks, where each memory' bank includes one or more memory' arrays. In some implementations, the PiM architecture 200 includes multiple bank groups, multiple memory’ die, or both. For example, a single memory’ die can include multiple bank groups and / or multiple bank groups can be distributed across multiple memory die.

[0034] The PiM architecture 200 includes multiple PiM blocks 202, where each PiM block 202 includes multiple compute elements. For example, the PiM block 202 can include discrete processors, processor units, register devices, buffers, and / or multiply accumulators (MACs) that cooperate to form one or more PiM compute elements. As discussed above, the PiM blocks 202 are used to execute computations for an example workload that originates at the SoC 102. The computations can be segmented into respective portions that are allocated between the SoC 102 and the memory device that includes the PiM blocks 202.

[0035] In the example of Fig. 2, a first PiM block 202 of PiM architecture 200 includes mode register 204-1 and process unit 206-1, whereas a second, different PiM block 202 of PiM architecture 200 includes mode register 204-2 and process unit 206-2. Each process unit 206-1, 206-2 can include a processor, a processor unit, or a processor core, such as a CPU. Each process unit 206-1. 206-2 can also include an example computation unit such as an arithmetic logic unit (ALU) or multiply-accumulate cell (MAC).

[0036] In some implementations, the PiM architecture 200 is included in the memory' device 122 as multiple discrete integrated circuits, where each integrated circuit is local to a given memory die (e.g., die-1 and die-2) and interacts or communicates with arrays of memory cells at that memory die. For example, the PiM architecture 200 can includecompute elements that are replicated and distributed across each of the memory die in the memory device 122. In some other implementations, the PiM architecture 200 is included in the memory device 122 as a single integrated circuit that interacts or communicates with each memory die of the memory device 122, including the arrays of memory cells at each memory die.

[0037] To boost PiM data access performance as described in this specification, the PiM blocks 202 or process units in the PiM architecture 200 are located within the memory device 122 but outside of a section of the memory device 122 that includes the bank groups. The section may be defined as a discrete memory die or defined in some other way (e.g., a portion of a memory die). Irrespective of the hardware configuration or layout of PiM architecture 200, the PiM blocks 202 are sufficiently external to the bank groups such that the PiM blocks 202 communicate with the bank groups based on a particular timing constraint that can be leveraged to boost PiM data access performance with cross bank group data aggregation.

[0038] The PiM operations can include standard CPU functions, whereas the CiM operations and MnC operations can include standard arithmetic operations, such as computations normally performed by an ALU or MAC. The CiM operations and MnC operations can also include computational functions of a HPU 114, such as multiplication and addition operations for matrix math, vector computations, linear algebra, and dot-product accumulations. In some implementations, each of the PiM operations, CiM operations, and MnC operations are performed in support of ML computations, neural network computations, or both.

[0039] In some implementations, the PiM operations are an extension of the computational functions of the HPU 114. For example, a PiM block 202 can generate accumulated values from sets of weight values / inputs and activation inputs obtained from memory banks of different bank groups based on a particular timing constraint. The timing constraint can be a minimum delay required between successive column commands issued to different bank groups within a DRAM device. The accumulated values are generated based on neural network computations performed using a computational array of the PiM block 202. The computational array can be a matrix multiplication unit with compute cells that are arranged as a systolic array. The accumulated values can be dot products of the sets of weight values and the activation inputs. That is, for a set of weights, the PiM block 202 multiplies each weight with each activation input and sums the products together to form an accumulated value.

[0040] The PiM architecture 200 can include a register or other portion of memory for storing data for a respective memory die or group of memory banks. For example, the data can be mode / configuration values. The data can also describe errors that occurred during a compute operation at a corresponding PiM block 202 of the memory device 122, or both. In some implementations, the register or other portion of memory is used to store configuration information, or associated instructions, for configuring aspects of a PiM block, or respective memory die, group of memory banks, or a combination of these.

[0041] For example, the mode registers 204-1, 204-2 can be used to control or trigger selection of a particular mode in a PiM architecture, such as an error-capture mode, interleave configuration mode, multi-batch processing mode, etc. In some implementations, a particular mode is selected based on bit values of the mode registers 204-1, 204-2. For example, to trigger or select an interleave configuration mode(s) or multi-batch processing mode(s), a single bit, or a sequence of bits, can be defined for use in the mode register.

[0042] Fig. 3 illustrates an example implementation of frequency voting for collaborative DVFS control at the system of Fig. 1. In the example of Fig. 3. the SoC 102 and memory device 122 can be configured, or combined, as an integrated circuit 300. For example, the SoC 102 and memory device 122 can be configured in a stacked configuration, in a single integrated circuit package, such that a bottom surface of the memory device 122 is mounted adjacent to a top surface of one or more circuit die that represent the SoC 102. In some implementations, the integrated circuit 300 is a sub-system of system 100 and is described alternatively as system 300.

[0043] Circuit / system 300 includes the SoC 102, memory' controller 105, and memory device 122 (described above), and a memory' physical layer (PHY) interface that is intermediate the memory controller 105 and memory device 122. As indicated at Fig. 3, a device of the SoC 102 can be configured as a host device (e.g., HPU 114) that communicates with the memory' device 122 to execute an example heterogeneous operation. For example, heterogeneous operations can include computations for vector math or matrix multiplication to execute certain ML workloads.

[0044] Each of the host device, system fabrics and memory controller 105. and memory PHY interface include and / or operate within a frequency & voltage domain of the system 100. Additionally, each of these components can be configured to support and / or implement DVFS controls and settings at the system 100. 300. The memory controller 105 uses the memory PHY interface to convey control and / or configuration signals 302. 304 from the SoC 102 to the memory device 122. System 100 includes control logic that implements a votingmethodology for establishing DVFS settings at the memory device 122. The voting methodology is implemented using the memory controller 105 of SoC 102.

[0045] The memory controller 105 uses the memory PHY interface to convey DVFS control and / or configuration signals 302, 304 from the SoC 102 to the memory device 122 in accordance with the voting methodology'. For example, the signals 302, 304 can include conventional votes for memory’ bandwidth and latency controls (302) and new votes for establishing PiM operating points (e.g., target operating points) based on the disclosed techniques (304). The DVFS control signals 304 can include operating frequency and operating voltage controls for configuring power output / consumption of the PiM block 202.

[0046] The memory device 122 processes signal communications from the SoC 102 in accordance with a clock signal 306. The signal communications can include controls, requests, and / or command signals from the SoC 102. For example, the request signals can also include or correspond to read / write commands for reading data from, or writing data to, memory’ cells / banks of the memory' device 122. Relatedly, and as indicated above, the signal communications also include DVFS control signals 304 that represent configuration commands from the SoC 102 for establishing certain DVFS settings at the memory- device 122.

[0047] The signal communications can include the clock signal 306. For example, the clock signal 306 can be a master or global DRAM clock signal that drives memory access and PiM operations at the memory device 122. The clock signal 306 can be passed to the memory device 122 as one or more of signals 302, 304. In some implementations, the clock signal 306 is generated by, or using, the SoC 102 and provided to the memory device 122 from the SoC 102. In some other implementations, the clock signal 306 is generated locally at the memory device 122 based on power signals routed via traces / lines of the integrated circuit 300. For example, the clock signal 306 may be generated independent of the SoC 102.

[0048] The memory device 122 can be configured to include or generate one or more local clock signals that trigger functions of components in the PiM architecture. The system 100, 300 can include an example clock divider circuit 308 that generates one or more local PiM clock signals 310. In some implementations, the PiM clocks (or clock signals) 310 are derived or generated based on clock signal 306. For example, the PiM clock signals 310 can be generated from a branch signal of a DRAM clock(s) of the memory device 122. In some implementations, memory device 122 includes an independent clock source that is local to one or more PiM blocks 202.

[0049] As indicated above, system 100 includes DVFS control logic for dynamically and collaboratively controlling the operating frequency and voltage (e.g., DVFS settings) of a host device and PiM block in an integrated random-access memory device 122 that couples to an SoC 102. For example, the DVFS control logic can be implemented and / or executed using the CPU 104, the PiM resource manager 108, or both. The CPU 104 and / or PiM resource manager 108 can execute the DVFS control logic in coordination with the memory controller 105.

[0050] In some implementations, the control logic is configured to determine a balance, or establish trade-offs, between overall system-level performance & power consumption of a host device and PiM block. In determining the balance, control logic evaluates and dynamically tune operating points to establish optimal performance & power ratios that allow for efficient processing of ML workloads while also satisfying user-experience criteria (described below). In the example of Fig. 3, the control logic can be used to establish operating frequencies corresponding to a high voltage value (VDD2H) 312 and a low voltage value (VDD2L) 314.

[0051] Fig. 4 illustrates example graphical data 400 indicating system-level operating points for a host device and a corresponding PiM block. The graphical data 400 includes sample operating points 402, each of which represents supported operating frequency levels of system 100. Hence, a target operating point that is established at the PiM block 202 can correspond to at least an operating frequency of the PiM block 202. the memory device 122. or both. Relatedly, a relationship between the operating frequency of the memory device 122 and the operating frequency of a PiM block 202 is characterized as a 1:1 mapping.

[0052] In some implementations, the graphical data 400 can be adapted such that the sample operating points 402 represent supported operating voltage levels of system 100 or a composite DVFS setting / level of system 100. The SoC 102 can compute multiple operating points of the memory device 122 and the PiM block 202, where each operating point of the multiple operating points corresponds to a distinct processing device of the SoC 102. For example, the SoC 102 can generate multiple sets of operating points 402 that span different device combinations, use case profiles / categories. user-experience criteria, frequency and voltages ranges, and ML model output types.

[0053] The system 100 can then determine all operating points 402 of a given host device and PiM block 202 that satisfy a given performance requirement. That is, the memory controller 105 can determine an optimal operating point 404, e.g., operating frequencies, of the host device and PiM block 202 to meet a performance target with the least amount ofenergy consumption and / or peak current. In some implementations, the CPU 104 and / or PiM resource manager 108 computes energy / power consumption estimates, generates a set of candidate host / PiM operating points based on the computed estimates, and determines host / PiM target operating points that have the least / lowest energy consumption among the candidate operating points. The energy consumption estimates can be system-level energyconsumption estimates generated by summing host device energy consumption and PiM block energy consumption.

[0054] In some implementations, the system 100 uses performance & power analytical models to process a multitude of data points to determine minimum thresholds for power efficient operating points. The threshold or target operating points can be selected that achieve the processing speeds required to meet performance targets that satisfy userexperience criteria, while also minimizing power consumption such that ML outputs are generated in a power efficient manner. The control logic of the SoC 102 selects a target operating point of the memory device 122 from among multiple operating points of the memory device 122 based on a voting methodology that is implemented using memorycontroller 105 (406).

[0055] In some instances, the PiM block 202 operating frequency can exceed an operating frequency of the host device based on votes from IP block 110 that requests a high DRAM operating point / frequency. In these instances, the memory controller 105 can: i) determine that the minimum operating frequency of the PiM block 202 exceeds an operating frequency of the host device; and ii) adjust the operating frequency of the host device to establish a minimum host operating frequency required to satisfy the performance target (408). The determination and / or adjustment can be also made using the CPU 104, the PiM resource manager 108. or both.

[0056] In some implementations, this frequency adjustment feature is executed using a feedback loop, or an interrupt or polling mechanism, that informs the host device (HPU 114) that an operating frequency of the PiM block 202 exceeds requested frequency or is operating at a high frequency threshold. The frequency adjustment feature can be implemented in hardware, software, or both.

[0057] Fig. 5 shows examples of user-experience criteria 502 for generating an output of a ML model and corresponding performance targets / requirements 504 that are required to satisfy a particular user-experience criteria. As indicated above, the hardware / software architecture dynamically balances the operating frequencies of a host device, such as HPU 1 14, and PiM block 202 for optimal energy efficiency within the established performancerequirements 504. In the example of Fig. 5, “tokens / sec” corresponds to a speed (or rate) of output generation from an ML model such as an LLM decoder.

[0058] The LLM decoder can include, or be configured to generate, a variety of output types 506, such as a text display output type, a speech sound output type, and a smart reply output type. In some implementations, the performance requirements 504 of an example LLM decoder workload varies based on use case categories and corresponding userexperience criteria for the categories, e.g.. speech output, text display, program coding, contents summarization, etc). The user-experience criteria 502 can include a minimum latency threshold for executing a ML task of a decoder of the LLM as well as a minimum time duration (e.g.. 1-5 second) for program code generation.

[0059] For example, the text display output type has a user-experience criteria that is based on an average human adult reading speed (508) and a corresponding performance target (or requirement) 514. In the example of Fig. 5, the indicated speed is 200-300 words per minute, which translates to a corresponding performance target (or requirement) of 4.44- 6.67 tokens / sec (514). In these and other examples, four tokens can be approximately three words in one or more of the common human languages.

[0060] In some implementations, the speech sound output type has a user-experience criteria that is based on an average human adult speaking speed (510) and a corresponding performance target 516. In the example of Fig. 5, the indicated speed is 100-150 words per minute, which translates to a corresponding performance target (or requirement) of 2.22-3.33 tokens / sec (516). The smart reply output type has a user-experience criteria that is based on an average visual latency perception for an human adult (512) and a corresponding performance target 518. For example, this human visual perception latency corresponds to the temporal delay in human visual systems (e.g., iris, lens, retina, cones, rods, etc.) that results in a latency between the occurrence of a visual stimulus (e.g., display of text on a device) and its perception (or comprehension) by the human observer. In the example of Fig. 5, the indicated latency is measured in milliseconds (ms) and translates to a corresponding performance target (or requirement) of 50 tokens / sec (518). In some examples, the latency can range from 50 ms to 500 ms.

[0061] Fig. 6 is an example process 600 for implementing collaborative system-level DVFS controls at the system 100. Process 600 is implemented or executed at system 100 using at least the SoC 102 and memory device 122 described above, including the integrated circuit 300 described with reference to Fig. 3. Hence, descriptions of process 600 will reference the above-mentioned computing resources of system 100. In some examples, thesteps or actions of process 600 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine- readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0062] Referring again to process 600, the system 100 identifies user-experience criteria for generating an output of a machine-learning (“ML”) model implemented at the integrated circuit (602). For example, the CPU 104 or host device can determine the user-experience criteria with reference to metadata derived from a model file generated by a compiler that compiles source code for the ML model. The model file can include the metadata along with a model binary (e.g., machine-code) that is executed by a host device of the SoC 102, such as the HPU 114 or GPU 118. In some implementations, the user-experience criteria are derived from a computational graph or simulation data of the ML model and with reference to processing capabilities of the host device, the SoC 102, or both.

[0063] The system 100 determines a performance target 504 required to satisfy the userexperience criteria (604). In particular, the SoC 102 computes or calculates a target performance of processing (e.g.. five tokens / sec or a specific time-to-first token) that is required to meet / satisfy a particular user-experience criteria (e.g., a number of words per minute) for a given output ty pe, such as text display. The performance target can be defined in two parts. For example, a first part of the performance target can be derived from the userexperience criteria, whereas a second part of the performance target can be DVFS settings or operating points that define a minimum operating frequency and voltage value.

[0064] In some implementations, the DVFS settings define minimum thresholds for power efficient operating points that achieve the processing, compute, and memory' access speeds required to generate ML outputs that satisfy certain user-experience criteria. The user-experience criteria can include a minimum number of output words per minute and the minimum number can be defined based on the average reading or speaking speed of an human adult. The output of the ML model can be generated from computations comprising matrix multiplications that are performed at the PiM block 202 using operands accessed from memory banks of the memory device 122.

[0065] The system 100 computes a target operating point of the memory device 122, where the target operating point is required to satisfy the performance target (606). In some implementations, computing a target operating point of the memory' device 122 includes determining a DVFS setting that: i) minimizes an operating frequency of the PiM block 202.ii) minimizes power consumption at the PiM block 202, and iii) satisfies the performance target for a given user-experience criteria.

[0066] The system 100 generates control signals that establish the target operating point at a PiM block 202 of the memory device (608). The system 100 can also compute a target operating point of the host device that is required to satisfy the performance target. The system 100 performs computations at the PiM block 202 using at least the target operating point established based on the control signals (610). The computations can be performed in response to memory-intensive data access operations that are executed to obtain operands stored at memory' banks of the memory' device 122.

[0067] In some implementations, each of the performance targets, the target operating point established at the PiM block 202, and the target operating point of the host device are used for dynamic voltage & frequency scaling at the integrated circuit. For example, the target operating point established at the PiM block can be a DVFS setting that includes a minimum operating frequency, voltage, and / or power output of the PiM block 202 required to satisfy the performance target.

[0068] Fig. 7 is a second example process for collaborative DVFS control at the system of Fig. 1. Similar to process 600, process 700 is also implemented or executed at system 100 using at least the SoC 102 and memory' device 122 described above, including the integrated circuit 300 described with reference to Fig. 3. In some examples, the steps or actions of process 700 are enabled or performed using programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine- readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0069] At process 700, the system 100, 300 calculates a target performance of processing required to meet (or satisfy ) a user experience criterion (702). For a data processing operation that involves an ML model, such as an LLM decoder, the target performance of processing can be a minimum speed (or rate) of output generation from the LLM decoder (e.g., tokens / sec, time-to-lst token, etc.). Relatedly, if the output being generated by the LLM decoder is speech sound, then an example user experience criterion can be the average human adult speaking speed of 100-150 w ords per minute. For this speech sound example, the system 100 can calculate that the target performance of processing is 2.22-3.33 tokens / sec to meet the user experience criterion of 100-150 words per minute for generating a speech sound output ty pe of the LLM decoder.

[0070] The system 100, 300 is configured to determine optimal operating frequencies of a host device 114 and a PiM block 202 to satisfy the target performance of processing with the least energy consumption, and / or peak current (704). For example, the system 100, 300 determines the minimum, and most power efficient, operating frequencies that are optimal for enabling the host device 114 and PiM block 202 to achieve the target performance for processing. In some implementations, the operating frequencies are the lowest and most power efficient DVFS settings that still allow the host device and PiM block to achieve the target performance of processing (e.g., 2.22-3.33 tokens / sec) to meet the user experience criterion of 100-150 words per minute for generating a speech sound output type of the LLM decoder.

[0071] The system 100, 300 determines these minimum power efficient operating frequencies and DVFS settings using at least the methodologies described above with reference to Fig. 4. In some implementations, the system 100, 300 leverages the disclosed techniques to determine a balance, or establish trade-offs, between the overall system-level performance & power consumption of a host device and PiM block. In determining the balance and / or establishing trade-offs, the collaborative DVFS control logic of the SoC 102 can evaluate and dynamically tune different sets of target operating points to establish optimal performance & power ratios that allow for efficient processing of ML workloads, while also satisfying user-experience criteria.

[0072] The system 100, 300 can compute multiple sets of target operating points and compute a corresponding power consumption value required to establish each target operating point. For instance, for each set of target operating points, the system 100, 300 can determine a respective operating frequency setting (e.g., 1.7 GHz) and voltage values (e.g., 1.05V) for the memory’ device 122 (or PiM block 202) and host device 114. For each of those sets, the system 100, 300 can then compute a corresponding power consumption value (e.g., 2m W) for establishing the operating frequency and voltage settings of that set.

[0073] The system 100, 300 activates a vote for the memory' operating frequency paired with the PiM frequency (706). In some implementations, the system 100, 300 is configured to activate the vote only during a time that the PiM block is active, such as when the PiM block is actively involved in computations for generating a model output. The system 100, 300 consolidates votes (or requests) to the memory-path frequency that are provided from various IP blocks of the SoC, and then selects the maximum vote among the votes (requests) (708). The maximum vote can be represented by a data value that specifies a maximum operating point requested by the IP block or an IP device of the IP block. For example, theoperating point specified by the maximum vote can be a maximum frequency (e.g., operating frequency) that is requested by an IP device of IP block 110, such as the host processing unit 114.

[0074] If the memory / PiM runs at a higher frequency than requested by the host device, for example, due to a vote from another IP block, then the system 100, 300 adjusts the frequency of the host device to a lower level, accordingly (710). For example, selection of the max vote may cause the operating frequency of the PiM block 202 to exceed an operating frequency of the host device. The higher PiM block frequency may be based on selection of a max vote from an IP device (e.g., GPU 118) of IP block 110 that requests a high DRAM operating point / frequency. The system 100. 300 is configured to adjust the operating frequency of the host device 114 to establish a minimum, power-efficient operating frequency that still allows the host device 114 to execute operations required to satisfy the target performance of processing.

[0075] The respective steps of process 600 and process 700 can be performed at a hardware integrated circuit as part of a larger compute operation to generate a machinelearning output, including an output for a neural network layer of a neural network that implements one or more ML models. For example, the output can be a portion of a computation for a ML task or inference workload to generate an image processing, speech processing, or image recognition output.

[0076] In some implementations, a portion of the integrated circuit can include a specialpurpose neural network processor or hardware ML accelerator configured to accelerate computations for generating different types of data processing outputs. In some implementations, one or more of the PiM operations, CiM operations, or MnC operations are performed by the memory device 122 to support or enable accelerating computations for generating different types of data processing outputs.

[0077] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[0078] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0079] The term "computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry', e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0080] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0081] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication netw ork.

[0082] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0083] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memon or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0084] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory7, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory7can be supplemented by, or incorporated in, special purpose logic circuitry7.

[0085] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid cry stal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0086] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matterdescribed in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0087] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0088] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0089] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0090] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0091] Further implementations are summarised in the following examples:

[0092] Example 1 : A method implemented using an integrated circuit comprising a System-on-Chip (“SoC”) and a memory device coupled to the SoC, the method comprising: identifying user-experience criteria for generating an output of a machine-learning (‘‘ML”) model implemented at the integrated circuit; determining a performance target required to satisfy one or more of the userexperience criteria; computing potential sets of operating points of a processing-in-memory ("PiM") block of the memory device and a host core of the SoC; selecting the most energy efficient set of operating points of the PiM block of the memory device and the host core; generating first control signals that establish a target operating point at the host core; generating second control signals that establish a target operating point at the PiM block of the memory device; and performing computations at the PiM block using the target operating point established based on the second control signals.

[0093] Computing the potential sets of operating points of the PiM and the host core may be based upon the performance target. Each of the potential sets of operating points may satisfy the performance target. The most energy' efficient set of operating points of the PiM block and the host core may be selected from the potential sets of operating points. The most energy efficient set of operating points may comprise the target operating point of the host core and the target operating point of the PiM block. The method may further comprise: performing computations at the host core using the target operating point established based on the first control signals.

[0094] Example 2: The method of Example 1. further comprising: generating the output consistent with the user-experience cnteria based on the target operating point established at the PiM block to satisfy the performance target.

[0095] Example 3: The method of Example 1 or 2, further comprising: computing, at the SoC, a target operating point of the host core that is required to satisfy the performance target.

[0096] Example 4: The method of Example 3, further comprising: generating the output consistent with the user-experience criteria by satisfying the performance target based on: i) the target operating point of the host core, and ii) the target operating point established at the PiM block.

[0097] Example 5: The method of any one of Examples 1 to 4, wherein the target operating point established at the PiM block corresponds to an operating frequency of the memory device.

[0098] Example 6: The method of Example 5, wherein a relationship between the operating frequency of the memory device and the operating frequency of the PiM block is characterized as a 1 : 1 mapping.

[0099] Example 7: The method of any one of Examples 1 to 6, further comprising: computing, at the SoC, a plurality of operating points of the memory device, wherein each operating point of the plurality' of operating points corresponds to a distinct processing device of the SoC.

[0100] Example 8: The method of Example 7. further comprising: selecting, at the SoC. the target operating point of the memory device from among the plurality of operating points of the memory device based on a voting methodology implemented using a memory controller of the SoC.

[0101] The voting methodology may be configured such that the selected target operating point of the memory device is a maximum operating point within the plurality of operating points.

[0102] Each of the plurality of operating points of the memory device may comprise an operating frequency of the memory device requested by a respective processing device of the SoC. The selected target operating point of the memory device may comprise a maximum operating frequency of the memory device within the operating frequencies of the memory device requested by the processing devices of the SoC.

[0103] Example 9: The method of any one of Examples 1 to 8, wherein the ML model is a large language model (“LLM") for a generative artificial intelligence (“‘Gen AT’) application.

[0104] Example 10: The method of Example 9, wherein the user-experience criteria include a minimum latency threshold for executing a ML task of a decoder of the LLM.

[0105] Example 11 : The method of Example 9 or 10, wherein the user-experience criteria include a minimum time duration defined based on human visual perception latency.

[0106] Example 12: The method of any one of Examples 9 to 11, wherein: i) the user-experience criteria include a minimum number of output tokens per minute; and ii) the minimum number is defined based on average human speaking speed or average human reading speed.

[0107] Example 13: The method of any one of Examples 1 to 12 as dependent from Example 3. wherein each of the performance target, the target operating point established at the PiM block, and the target operating point of the host core are used for dynamic voltage & frequency scaling (“DVFS”) at the integrated circuit.

[0108] Example 14: The method of any one of Examples 1 to 13, wherein the target operating point established at the PiM block is a DVFS setting comprising a minimum operating frequency of the PiM block that is required to satisfy the performance target.

[0109] Example 15: The method of Example 14, further comprising: determining that the minimum operating frequency of the PiM block exceeds an operating frequency of the host core; and adjusting the operating frequency of the host core to establish a minimum host operating frequency required to satisfy' the performance target.

[0110] Example 16: The method of any one of Examples 1 to 15, further comprising: computing a target operating point of the memory device by at least: determining a DVFS setting that: i) minimizes an operating frequency of the PiM block, ii) minimizes power consumption at the PiM block, and iii) satisfies the performance target.

[0111] Example 17: An integrated circuit comprising: a System-on-Chip (“SoC”); a memoty device coupled to the SoC; and a processor and a non-transitoiy machine-readable storage medium for storing instructions that are executable by the processor to cause performance of operations comprising: identifying user-experience criteria for generating an output of a machinelearning (“ML”) model implemented at the integrated circuit; determining a performance target required to satisfy one or more of the userexperience criteria; computing potential sets of operating points of a processing-in-memory (“PiM”) block of the memory device and a host core of the SoC;selecting the most energy efficient set of operating points of the PiM block of the memory device and the host core; generating first control signals that establish a target operating point at the host core; generating second control signals that establish a target operating point at the PiM block of the memory device; and performing computations at the PiM block using the target operating point established based on the second control signals.

[0112] Optional features of Example 1 (e.g., Examples 2 to 16) also apply to Example 17.

[0113] Example 18: A method implemented using an integrated circuit comprising a System-on-Chip (“SoC”) and a memory device coupled to the SoC, the method comprising: identifying user-experience criteria for generating an output of a machine-learning (“ML”) model implemented at the integrated circuit; determining a performance target required to satisfy one or more of the userexperience criteria; computing: i) a processing-in-memory (“PiM”) target operating point of the memory device and ii) a host target operating point of a host core of the SoC, wherein each of the PiM and host target operating points define a minimum, power efficient measure of performance that is required to satisfy the performance target; generating control signals that establish: i) the PiM target operating point at a PiM block of the memory' device and ii) the host target operating point at the host core; and performing computations at: i) the PiM block using the PiM target operating point and ii) the host core using the host target operating point established based on the control signals.

[0114] Further or alternatively, the PiM and host target operating points may collectively define a combined minimum, power-efficient measure of performance of the PiM block and the host core that is required to satisfy' the performance target.

[0115] Example 19: The method of Example 18, further comprising: computing a first plurality of sets of operating points for the memory device; selecting a particular set of operating points from the first plurality of sets of operating points as the PiM target operating point; computing a second plurality of sets of operating points for the host core; and selecting a particular set of operating points from the second plurality of sets of operating points as the host target operating point.

[0116] Example 20: The method of Example 19, further comprising: evaluating trade-offs in performance output and power consumption between respective sets of operating points that are computed for the memory device.

[0117] Example 21 : The method of Example 19 or 20, further comprising: evaluating trade-offs in performance output and power consumption between respective sets of operating points that are computed for the host core.

[0118] Example 22: The method of any one of Examples 19 to 21, wherein: each set of operating points in the first plurality of sets of operating points has a different power consumption or energy efficiency from other sets in the first plurality of sets; and each set of operating points in the second plurality of sets of operating points has a different power consumption or energy efficiency from other sets in the second plurality of sets.

[0119] Example 23: The method of any one of Examples 19 to 22, further comprising: computing a respective energy consumption estimate for each set in the first plurality of sets of operating points for the memory device; determining a PiM minimum / least energy consumption among the respective energy7consumption estimates computed for each set in the first plurality of sets of operating points; and establishing a system-level energy consumption using the PiM minimum / least energy consumption.

[0120] Example 24: The method of Example 23, further comprising: computing a respective energy consumption estimate for each set in the second plurality of sets of operating points for the memory device; determining a host core minimum / least energy7consumption among the respective energy consumption estimates computed for each set in the second plurality of sets of operating points; and establishing the system-level energy consumption using the PiM minimum / least energy consumption and the host core minimum / least energy consumption.

[0121] Any optional features of Example 1 (e.g., any of Examples 2 to 16) also apply to Example 18. Any optional features of Example 18 (e g., any of Examples 19 to 24) also apply to Example 1.

[0122] Example 25: The method of any one of Examples 1 to 16 and 18 to 24, further comprising: generating the output of the ML model based upon an input sample, wherein generating the output comprises performing the computations at the PiM block using the target operating point established at the PiM block and / or at the host core using the target operating point established at the host core, and the output comprises data associated with at least one of: a text display: a speed sound: a smart reply; and a program code generation.

[0123] Example 26: The method of Example 25, wherein the input sample is derived from at least one of: image data; speech data; and a text corpus.

Claims

What is claimed is:

1. A method implemented using an integrated circuit comprising a System-on-Chip (“SoC”) and a memon device coupled to the SoC, the method comprising: identifying user-experience criteria for generating an output of a machine-learning ("ML") model implemented at the integrated circuit; determining a performance target required to satisfy one or more of the userexperience criteria; computing potential sets of operating points of a processing-in-memory (“PiM”) block of the memory device and a host core of the SoC; selecting the most energy efficient set of operating points of the PiM block of the memory device and the host core; generating first control signals that establish a target operating point at the host core; generating second control signals that establish a target operating point at the PiM block of the memory device; and performing computations at the PiM block using the target operating point established based on the second control signals.

2. The method of claim 1 , further comprising: generating the output consistent with the user-experience criteria based on the target operating point established at the PiM block to satisfy the performance target.

3. The method of claim 1 or 2, further comprising: computing, at the SoC. a target operating point of the host core that is required to satisfy the performance target.

4. The method of claim 3, further comprising: generating the output consistent with the user-experience criteria by satisfying the performance target based on: i) the target operating point of the host core, and ii) the target operating point established at the PiM block.

5. The method of any preceding claim, wherein the target operating point established at the PiM block corresponds to an operating frequency of the memory device.

6. The method of claim 5, wherein a relationship between the operating frequency of the memory device and the operating frequency of the PiM block is characterized as a 1 : 1 mapping.

7. The method of any preceding claim, further comprising: computing, at the SoC. a plurality of operating points of the memory device, wherein each operating point of the plurality of operating points corresponds to a distinct processing device of the SoC.

8. The method of claim 7, further comprising: selecting, at the SoC, the target operating point of the memory device from among the plurality of operating points of the memory device based on a voting methodology implemented using a memory7controller of the SoC.

9. The method of any preceding claim, wherein the ML model is a large language model (‘ LLM”) for a generative artificial intelligence (“GenAI”) application.

10. The method of claim 9, wherein the user-experience criteria include a minimum latency threshold for executing a ML task of a decoder of the LLM.

11. The method of claim 9 or 10, wherein the user-experience criteria include a minimum time duration defined based on human visual perception latency.

12. The method of any one of claims 9 to 11, wherein: i) the user-experience criteria include a minimum number of output tokens per minute; and ii) the minimum number is defined based on average human speaking speed or average human reading speed.

13. The method of any preceding claim as dependent from claim 3, wherein each of the performance target, the target operating point established at the PiM block, and the target operating point of the host core are used for dynamic voltage & frequency scaling (“DVFS”) at the integrated circuit.

14. The method of any preceding claim, wherein the target operating point established at the PiM block is a DVFS setting comprising a minimum operating frequency of the PiM block that is required to satisfy the performance target.

15. The method of claim 14, further comprising: determining that the minimum operating frequency of the PiM block exceeds an operating frequency of the host core; and adjusting the operating frequency of the host core to establish a minimum host operating frequency required to satisfy' the performance target.

16. The method of any preceding claim, further comprising: computing a target operating point of the memory device by at least: determining a DVFS setting that: i) minimizes an operating frequency of the PiM block, ii) minimizes power consumption at the PiM block, and iii) satisfies the performance target.

17. An integrated circuit comprising: a System-on-Chip (“SoC”); a memory device coupled to the SoC; and a processor and a non-transitoiy machine-readable storage medium storing instructions that are executable by the processor to cause performance of operations comprising: identifying user-experience criteria for generating an output of a machinelearning ('‘ML’’) model implemented at the integrated circuit; determining a performance target required to satisfy one or more of the userexperience criteria; computing: i) a processing-in-memory (“PiM") target operating point of the memory device and ii) a host target operating point of a host core of the SoC. wherein each of the PiM and host target operating points define a minimum, power efficient measure of performance that is required to satisfy the performance target;generating control signals that establish: i) the PiM target operating point at a PiM block of the memory device and ii) the host target operating point at the host core; and performing computations at: i) the PiM block using the PiM target operating point and ii) the host core using the host target operating point established based on the control signals.

18. The integrated circuit of claim 17, wherein the operations further comprise: computing a first plurality of sets of operating points for the memory device; selecting a particular set of operating points from the first plurality of sets of operating points as the PiM target operating point; computing a second plurality’ of sets of operating points for the host core; and selecting a particular set of operating points from the second plurality of sets of operating points as the host target operating point.

19. The integrated circuit of claim 18. wherein the operations further comprise: evaluating trade-offs in performance output and power consumption between respective sets of operating points that are computed for the memory device.

20. The integrated circuit of claim 18 or 19, wherein the operations further comprise: evaluating trade-offs in performance output and power consumption between respective sets of operating points that are computed for the host core.

21. The integrated circuit of any one of claims 18 to 20. wherein: each set of operating points in the first plurality of sets of operating points has a different power consumption or energy efficiency from other sets in the first plurality of sets; and each set of operating points in the second plurality of sets of operating points has a different power consumption or energy efficiency from other sets in the second plurality of sets.

22. The integrated circuit of any one of claims 18 to 21, wherein the operations further comprise:computing a respective energy consumption estimate for each set in the first plurality of sets of operating points for the memory device; determining a PiM minimum / least energy consumption among the respective energy consumption estimates computed for each set in the first plurality of sets of operating points; and establishing a system-level energy consumption using the PiM minimum / least energy consumption.

23. The integrated circuit of claim 22, wherein the operations further comprise: computing a respective energy consumption estimate for each set in the second plurality of sets of operating points for the memory device; determining a host core minimum / least energy consumption among the respective energy consumption estimates computed for each set in the second plurality of sets of operating points; and establishing the system-level energy consumption using the PiM minimum / least energy consumption and the host core minimum / least energy consumption.

Images