Dynamic task scheduling based on runtime conditions

Abstract

Methods and systems, including computational instructions / programs encoded on computer-readable media, are described for dynamically allocating tasks related to a workload to multiple processors based on runtime information. One example method can be performed using a System-on-Chip ("SoC"). The example method can include obtaining a first task allocation that allocates a first set of tasks between a first processor of the SoC and a second processor external to the SoC, identifying runtime information determined upon execution of a subset of the first set of tasks, adjusting the first task allocation based on the runtime information to obtain a second task allocation that allocates remaining tasks in the first set of tasks, and executing one or more tasks based on the second task allocation.

Description

DYNAMIC TASK SCHEDULING BASED ON RUNTIME CONDITIONSBACKGROUND

[0001] This specification generally relates to dynamic task scheduling in computing systems having multiple processing units.

[0002] Computing systems, which can include edge devices (e.g., mobile phones, tablets, etc.), can include multiple processing units, such as a central processing unit (‘’CPU”), a host processing unit (“HPU”), and / or a processor-in-memory (“PIM”) unit. A HPU can be designed to perform specialized tasks, such as tasks related to handling of machine learning workloads (which can include, e.g., matrix multiplication operations). In some instances, the same workload can be executed on other processing units of the computing system, such as on the PIM unit. Each of these processing units can have different operational characteristics (e.g., performance and energy consumption characteristics), such that their performance characteristics (e.g., execution time, energy' consumption) may vary when executing the same (or similar) workloads.SUMMARY

[0003] Computing systems that execute tasks related to machine learning workloads can include multiple processors. Subsets of tasks related to a machine learning workload can be allocated to one or more processors depending on the nature of the tasks and characteristics of each particular processor. For example, a HPU can be designed for large- scale parallel execution of matrix operations. As another example, a PIM is designed to provide a compute resource with a dedicated memory with high memory bandwidth for memory-intensive tasks. Thus, in some cases, some tasks of the machine learning workload are allocated to a first processor, e.g., an HPU, and other tasks are allocated to a second processor, e.g.. a PIM block. As described in this document, runtime information about the processors as they execute a workload can be used to dynamically adjust the allocation of tasks between processors, to meet predetermined system scheduling and / or performance criteria.

[0004] In some implementations, the techniques include a method performed using a System-on-Chip (“SoC”), the method including at least the operations of obtaining a first task allocation that allocates a first set of tasks between a first processor of the SoC and a second processor external to the SoC; identifying runtime information determined upon execution of a subset of the first set of tasks; adjusting the first task allocation based on the runtimeinformation to obtain a second task allocation that allocates remaining tasks in the first set of tasks; and executing one or more tasks based on the second task allocation.

[0005] In some implementations, the first processor can be a machine-learning (ML) hardware accelerator of the SoC and the second processor can be a processing unit in an integrated memory device coupled to the SoC.

[0006] In some implementations, the methods can further include the operations of obtaining, at compile time, a set of estimated operational parameters for each of the first processor and the second processor; and computing, for each of the first processor and the second processor, a set of estimated performance metrics using the estimated operational parameters for the respective processor.

[0007] In some implementations, the first task allocation can be based on the sets of estimated operational parameters and the sets of estimated performance metrics.

[0008] In some implementations, the runtime information can include a first set of runtime parameter values that are generated upon the first processor executing tasks allocated according to the first task allocation: and a second set of runtime parameter values that are generated upon the second processor executing tasks allocated according to the first task allocation.

[0009] In some implementations, adjusting the first task allocation based on the runtime information can include adjusting a set of estimated performance metrics for the first processor using the first set of runtime parameter values and the set of estimated performance metrics for the first processor, to obtain a set of adjusted performance metrics for the first processor; adjusting a set of estimated performance metrics for the second processor using the second set of runtime parameter values and the set of estimated performance metrics for the second processor, to obtain a set of adjusted performance metrics for the second processor; identifying, using control information, a particular adjusted performance metric from each set of adjusted performance metrics; comparing the identified particular adjusted performance metrics; and based on the comparison of the identified adjusted performance metrics, adjusting the first task allocation to obtain the second task allocation.

[0010] In some implementations, the control information can specify a target metric to be evaluated for allocating tasks between the first processor and the second processor.

[0011] In some implementations, the second task allocation can allocate the remaining tasks differently between the first processor and the second processor relative to the first task allocation.

[0012] In some implementations, the first task allocation can allocate the first set of tasks to the first processor, and the second task allocation can allocate a first subset of the remaining tasks to the first processor and a second subset of the remaining tasks to the second processor.

[0013] In some implementations, executing one or more tasks in the remaining tasks based on the second task allocation can include executing, by the first processor and according to the second task allocation, the first subset of the remaining tasks; and executing, by the second processor and according to the second task allocation, the second subset of the remaining tasks.

[0014] In some implementations, each of the SoC and the second processor can be implemented at the same edge device.

[0015] In some implementations, the techniques include another method performed using a System-on-Chip (“SoC”), the method including at least the operations of generating a first allocation of tasks on a System-on-Chip (“SoC”) of a device; identifying runtime information on the device; generating a second allocation of tasks by adjusting the first allocation of tasks based on the runtime information; and executing a task provided by the second allocation of tasks on a Processor-in-Memory (“PIM”) block of an integrated memory device coupled to the SoC.

[0016] Other implementations of these and other aspects and techniques summarized above, include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0017] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Fig. 1 is a block diagram of an example computing system for performing compiler operations.

[0019] Fig. 2 is a block diagram of an example computing system with at least one SoC.

[0020] Fig. 3 illustrates an example process for adjusting a task schedule based on runtime conditions.

[0021] Fig. 4 is a flow diagram of an example process for adjusting an allocation of tasks.

[0022] Fig. 5 is a flow diagram of an example process for adjusting an allocation of tasks.

[0023] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0024] This specification generally relates to computing systems that leverage runtime information to dynamically allocate computing tasks between different processors of such systems.

[0025] As summarized here and described below in greater detail with reference to FIGS. 1-5, a computing system can be configured to process a particular task workload (e.g., a machine learning (ML) workload) using the multiple processing units — e.g., a host processing unit (HPU) and a processing-in-memory (PIM) unit — deployed in such systems. The HPU can be configured to process particular types of workloads, such as, e.g., ML workloads that involve matrix processing operations. A PIM can be configured to process the same (or similar) workloads. The HPU can perform better than the PIM when confronted with workloads that have a low data access rate and high compute rate, whereas the PIM can perform better than the HPU for processing workloads that have relatively higher memory bandwidth and energy consumption needs. For ease of description and brevity, the following description is provided in the context of allocating workloads between the HPU and the PIM processing unit. However, one skilled in the art will appreciate that the same techniques described are applicable to other types of processing units and other types of workloads (e.g., workloads other than ML workloads) that the processing unit may be specifically configured to process. One skilled in the art will also appreciate that the described dynamic allocation of tasks related to a workload need not be limited to an allocation between two processors (e.g., may be allocated between more than two processors).

[0026] In the example use case of tasks related to a ML workload that may be processed by a computing system having at least a first processor and a second processor (e.g., a HPU and a PIM), the computing system can include processing logic that estimates each processing unit’s operational characteristics or parameters (e.g., execution time, energy consumption, local temperature, other operational details such as operating voltage, frequency, available memory bandwidth) for processing these tasks. A compiler can generate binaries for the HPU and the PIM. and add, to each respective binary, metadata including the estimated operational characteristics for processing the tasks by each respective processing unit.

[0027] A scheduler (e.g.. processing logic implemented by the HPU) can schedule or allocate tasks between the HPU and the PIM based on the estimated operational parameters, e.g., included in the metadata. For example, the scheduler can allocate all tasks to be performed by the HPU or can allocate some tasks to be performed by the HPU while other tasks performed by the PIM.

[0028] At runtime, an initial set of tasks of the workload can be executed by the HPU and / or the PIM per the scheduler’s allocation of tasks. The processing logic can determine the runtime information for each processing unit (e.g., temperature, operating frequency, voltage, throughput, available memory. etc.) from the execution of these initial tasks. In some cases, the realized runtime operational parameters can be different from the expected operational parameters estimated at compile time. Depending on the system hardware and software conditions during runtime, the runtime information may be different from the default information used for the first task allocation (at compile time). Indeed, some information and parameters may only be available during runtime.

[0029] An adjustor, which can be implemented as processing logic executed by a processing unit of the computing system, e.g., the HPU, can adjust the estimated operational parameters for each processing unit based on this runtime information. Additionally, based on predetermined control criteria (e.g., performance or energy consumption) and the adjusted values computed for each processing unit, the scheduler can determine to allocate the remaining tasks between HPU and the PIM differently than how those tasks were initially allocated. For example, if a system’s control criteria indicate higher preference for performance, the scheduler may reallocate some tasks to the PIM upon determining that the adjusted performance (e.g., adjusted time) is lower for the PIM relative to the HPU. In this example, if all the tasks were initially allocated to the HPU. the scheduler may determine to reallocate tasks such that some proportion of the remaining tasks would be executed by theHPU. with the remaining tasks being executed by the PIM. In some implementations, the runtime assessments of each processor can be done periodically, e.g.. after completion of a certain group of tasks or after a certain amount of processing time has elapsed. In this manner, the system described herein can use runtime information to dynamically and iteratively update an allocation of tasks of a workload between two (or more) processors.

[0030] These and additional features are described now with reference to FIGS. 1-5.

[0031] Fig. 1 is a block diagram of an example computing system 100 for performing operations relating to a dynamic allocation of tasks between processors of the computing system 100 based on runtime information.

[0032] The system 100 includes a compiler 102 configured to generate executable code, which can include assembly code, machine code, object code, or other code corresponding to an example low-level programming language. Compiler 102 can be an example program that translates computer code written in one programming language (the source language) into another language (the target language). In general, compiler 102 translates source code from a high-level programming language to a low-level programming language, such as assembly or machine code, to create an executable program.

[0033] In the example of system 100 of Fig. 1, the compiler 102 is integrated in, or accessible by, an example computing device 107, such as a consumer electronic device or mobile / client device. In some implementations, computing device 107 is represented by example items such as tablets, laptops, Chromebooks. eNotebooks. Netbooks, or other related mobile computers. In some implementations, compiler 102, and input data 104 (including source code 106 and metadata 114), are accessed using a desktop computer, network server, or related cloud-based asset.

[0034] The compiler 102 generates executable code based on compiler operations performed on input data 104. The input data 104 can include program / source code 106 (‘’source code 106”) written using a high-level programming language. In the system 100 of FIG. 1, the input data 104 includes source code corresponding to a neural network model. The input data 104 can also include metadata 114. The metadata 114 can include one or more compile-time assumptions of expected runtime conditions (also referred to herein as runtime parameters) of one or more processors, e.g., of a neural network processor which is an integrated circuit 134 installed on a SoC, a processor of a PIM block, etc.). The expected runtime conditions include, e.g., expected environmental and operational conditions of the processor(s) that execute the executable code generated by the compiler 102. For example, the metadata 114 can include one or more of temperature, memory bandwidth, availablecompute throughput, operating voltage, operating frequency, and other operational parameters that can affect performance of a processor (e.g., the integrated circuit 134) that executes the executable code generated by the compiler 102.

[0035] The compiler 102 generates the executable code in response to compiling the source code 106 based in part on a set of compiler operations 112. The compiler 102 includes the metadata 114 in a binary file that includes the executable code.

[0036] In some implementations, the compiler operations 112 are a set of core compiler operations generally (or normally) performed by some, most, or any example compiler(s).

[0037] For instance, the compiler operations 112 can include multiple compiler phases or stages, such as lexical analysis, syntactic analysis and / or parsing, semantic analysis, and code generation, including intermediate code generation and / or code optimization. In the example of Fig. 1, compiler 102 can be a special -purpose program that translates source code 106 into machine code, bytecode, or another machine-readable programming language. The source code 106 can be written in an example high-level, human-readable programming language such as Python, Java, or C++.

[0038] In some implementations, the source code 106 is example code generated using platforms such as TensorFlow®, OpenAI®, or PyTorch®. In the context of machine learning, the compiler 102 is configured to generate executable code in the form of a model binary 110 that is used to execute an example machine-learning (“ML”) model. For example, the ML model can be based on a multi-layer neural network. The multi-layer neural network is a software construct. Thus, the neural network can be any type of artificial (e.g., non- organic) neural network, such as a convolutional neural network (“CNN”) or a recurrent neural network (“RNN”).

[0039] The system 100 also includes a performance estimator 116 that can be implemented, e.g., at a computing device 107. The performance estimator 116 can process the metadata 114 and the source code 106 to estimate performance metrics (e.g., execution time (latency), power performance) for the processor (e.g., integrated circuit 134) under the estimated runtime conditions represented by the metadata 114 — thereby estimating the performance of the processor for executing the executable code generated by the compiler 102. For example, the set of estimates represented by the metadata 114 along with the source code 106 are used to evaluate one or more target metrics including a performance target (e g., execution time (latency), temperature, memory bandwidth, operating voltage, operating frequency, etc.), energy / power consumption of a particular device that executes the modelbinary' 110. and / or a weighted sum of the performance target and the energy / power consumption target.

[0040] In some implementations, the performance metrics can be estimated by the performance estimator 116 using an analytical modeling approach. In such implementations, a dedicated analytic model is generated for a respective processor, and application level parameters are provided to such a model, which are then used to estimate the performance metrics. In this example, the performance estimator 116 obtains an analytical model for a particular processor (e.g., host device on the SoC, PIM, etc.), which is specific to each device. The performance estimator 116 determines operational and environmental parameters (e.g., temperature, operating voltage, memory bandwidth, etc.) based on an expected environment and device specifications and evaluates the analytical model as a function of the parameters and workload, to generate one or more estimated performance metrics.

[0041] In some implementations, application parameters include a number of operations performed by a particular processor and a number of accessed bytes from a memory. The performance estimator 116 can use the application parameters to estimate performance and energy consumption of the particular processor. For example, the performance estimator 116 can estimate a compute performance of the particular processor by dividing the number of operations by a compute throughput (operations / second) to estimate a compute time. If performance is bound by memory accesses, the performance estimator 1 16 can divide the number of accessed bytes from memory by a memory bandwidth (GB / s) to estimate a memory-accessed time. As another example, the performance estimator 116 can estimate energy consumption of the particular processor by multiplying the number of operations and the number of accessed bytes from memory by pJ (pico joules) / operation and pJ / byte, respectively.

[0042] In general, analytical models that estimate performance and / or energy consumption related to executing a series of operations include one or more assumptions about operational and environmental parameters of a respective processor. For example, an analytical model that assumes a particular memory bandwidth is available during runtime, may assume that other parameters that can include no contention with other processes, room temperature operation, etc.

[0043] One example of an analytical model is a roofline model that can be implemented to estimate performance of a processor. As a measure of processor performance, an execution time can be evaluated as the slower of a data transfer time and a compute time. A system that includes data transfer and data processing is limited by the slower of the datatransfer and the compute. The data transfer time can be further represented as a ratio of data traffic in terms of bytes read from memory and memory bandwidth in terms of bytes per second. The compute time can be further represented as a ratio of a total number of operations and a compute throughput in terms of operations per second. To illustrate how an execution time, e.g., an example of a performance metric of a processor, is affected by an operational parameter, consider a case in which available data transfer bandwidth of an HPU is halved due to contention with other concurrent tasks, a memory and compute throughput of the HPU is modified by a factor of 1.2x due to frequency scaling. An updated execution time can be estimated as the slower of the estimated data transfer time multiplied by 2 and the estimated compute time divided by 1.2. In addition, other analytical methods for adjusting one or more performance parameters are possible depending on the specific components and functionality of the computing system.

[0044] Additionally, or alternatively, the performance metrics can be generated by the performance estimator 116 using a profiling methodology, whereby target workloads (similar to the workloads expected to be executed in an actual operation) are executed by a particular processor and performance metrics are measured in response to execution of such workloads. In some implementations, the performance estimator 116 performs a profiling operation in relation to the processor that executes the model binary7. In this example, the performance estimator 116 includes a compiler (e.g., a compiler similar to the compiler 102) or performs a compilation step using the compiler 102 to compile the source code 106 and execute a respective model binary (e.g., a model binary similar to the model binary 1 10) on the processor being evaluated one or more times. The performance estimator 116 estimates performance metrics from the iterative execution of target workloads by the processor for which the estimation is being performed.

[0045] In this manner, the outputs of both the analytical approach and the profiling approach (described above) generate estimates of performance metrics and energy consumption metrics associated with the processor that executes the model binary7under operational parameters that affect the processor performance (e.g., temperature, operating voltage, memory bandwidth, etc.).

[0046] In some implementations, the performance estimator 116 determines performance metrics during the profiling operations to evaluate an execution time of workloads, e g., a series of operations. The performance estimator 116 evaluates execution times by using hardware timers of a processor, e.g., the HPU. For example, the system can reset a hardware timer at the beginning of a workload execution. After each clock cycle of theprocessor, the hardware timer counts up until the workload execution is complete. The processor can read the counter and multiply the value by the clock-cycle interval to determine an end-to-end execution of the workload.

[0047] The estimated performance metrics along with the metadata 114 can be provided by the performance estimator 116 to the compiler 102, which can format this data as compiler hints / metadata tags 108.

[0048] The compiler 102 executes the compiler operations 112 that convert the source code 106 into executable code in the form of the model binary 110. The compiler 102 also combines the generated model binary with the computed compiler hints / metadata tags 108, e.g., by appending the compiler hints / metadata tags 108 to the generated model binary 110.

[0049] In some implementations, the compiler 102 executes the compiler operations 1 12 on the input data 104, to generate a separate binary for each processor that is expected to execute the binary. For example, the compiler 102 can generate a first model binary for a first processor (e.g., a host processor unit of the SoC) and append thereto compiler hints / metadata tags 108 corresponding to that processor. Similarly, the compiler 102 can generate a second model binary for a second processor (e.g.. a PIM block external to the SoC) and append thereto compiler hints / metadata tags 108 corresponding to that processor. In some cases, the metadata 114 associated with each device is different. For example, memory7bandwidth may be larger for the PIM block compared to the processor on the SoC and the operating frequency for the PIM block may be smaller compared to the processor on the SoC.

[0050] In this manner, the compiler 102 generates two model binaries, each with a set of compile-time estimates of operational parameters and performance metrics (e.g., latency, throughput, temperature, memory bandwidth, operating frequency, energy consumption, execution time, etc.) corresponding to a respective processor (e.g., the host processing unit or the PIM block).

[0051] In some implementations, based on the one or more performance metrics obtained from the performance estimator 116, a scheduler (e.g., one or more programming instructions executing on a processor, e.g., the host processing unit) generates a task allocation for a first set of tasks. For example, the task allocation indicates an allocation of the first set of tasks for execution by a first processor, e.g., a processor on the SoC such as a host processing unit (also referred to above as the integrated circuit 134). As another example, the task allocation can indicate an allocation of the first set of tasks for execution by multiple processors, e.g., a first subset of tasks executed by the processor on the SoC and asecond subset of tasks executed by a second processor external to the SoC (e.g., the PIM implemented at a memory device).

[0052] In some implementations, the model binary 110 generated by the compiler 102 is loaded and executed by one or more processors implemented on a target user device, e.g., a target user device of a set of target user devices 136. Each target user device includes an SoC and / or a PIM block, as described in relation to Fig. 2 below.

[0053] Fig. 2 is a block diagram of an example computing system 200 that includes a system-on-chip 202 (“SoC 202”). In some implementations, the computing system 200 is implemented on a user / client device 230, similar to the target user devices 136 described in relation to FIG. 1. The SoC 202 includes a central processing unit 204 (“CPU 204”), a memory controller 205, a shared memory 206 (“memory 206”), a workload task scheduler 208, and an IP / circuit block 210. In some implementations, system 200 can include multiple SoCs and any descriptions for the SoC 202 herein are equally applicable to each of the multiple SoCs that may be included at system 200.

[0054] The memory 206 can be a system memory, shared memory, or both. In the example of Fig. 2, memory 206 is depicted external to the IP / circuit block 210. However, memory 206 can include portions of memory that are: i) specific to circuit block 210, ii) external to the IP / circuit block 210, or iii) both. The memory' 206 can be random access memory of the SoC 202, such as static random access memory (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0055] In some implementations, aspects of memory 206 are configured as a shared scratchpad memory' that supports parallel access of its memory resources by two or more processors of the IP / circuit 210. Memory 206 can also include various other types of memory, such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0056] The workload task scheduler 208 is implemented in hardware and software. Aspects of the workload task scheduler 208 can be also implemented as firmware of the SoC 202 or firmware of a device of the SoC 202. such as a DRAM memory device or the CPU 204. The workload task scheduler 208 includes control logic implemented in hardware, software, or both. For example, the workload task scheduler 208 can include resources such as flip-flops, registers, buffers, etc., that are implemented in hardware and control logic (e.g., programmed code) that is implemented in software.

[0057] The IP / circuit block 210 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the IP / circuit block 210 can include an image signal processor (ISP) 212, a host processing unit (HPU) 214, a digital signal processor (DSP) 216, and a graphics processing unit (GPU) 218. The IP / circuit block 210 is referred to alternatively as an IP block 210, where the IP block can include one or more hardware elements. For example, each of the ISP 212, HPU 214, DSP 216, and GPU 218 can be a respective IP block (or IP device) of a particular entity or device manufacturer.

[0058] The HPU 214 can be a special-purpose processor, such as machine-learning hardware accelerator, neural processor unit, neural network processor, or an applicationspecific processor. In some implementations, the HPU 214 is a neural network tensor processor that includes an integrated circuit architecture configured for power-efficient execution of machine-learning computations using tensor constructs such as multidimensional matrices.

[0059] One or more aspects of the workload task scheduler 208 can be implemented as a software routine of the CPU 204, which uses one or more hardware resources of the CPU 204, such as registers, buffers, etc. The CPU 204 can be configured as an instruction and vector data processing engine that processes data obtained from a system memory of the SoC 202, such as memory 206. In some implementations, each processor (e.g., ISP 212, DSP 216, HPU 214, GPU 218) of the SoC 202 includes multiple cores. The workload task scheduler 208 can generate control signaling 224 to manage and distribute compute operations to a memory7device 222 (e.g., DRAM) or an HPU 214. The control signaling 224 is routed in system 200 using an example bus 220 of the SoC 202. The control signaling 224 can include commands, requests, data, instructions, or some combination of these.

[0060] The workload task scheduler 208 cooperates with the CPU 204 and memory controller 205 to dynamically control and manage scheduling of one or more compute operations. In some implementations, the compute operations are executed at the SoC 202 in support of a heterogeneous compute operation between two or more processing units that are included among the IP block 210, the CPU 204. or both. Alternatively, one or more aspects of the workload task scheduler 208 can be implemented as a software routine of the host processing unit 214.

[0061] The system 200 includes a memory' device 222. The memory' device 222 can include multiple memory’ dies. For example, the memory device 222 can include N memory die, where N is an integer greater than 1 . The memory’ device 222 can be a dynamic random-access memory (DRAM) or Double Data Rate (DDR) synchronous DRAM (SDRAM). The memory device 222 is configured to perform or support various types of PIM operations and memory-near-computing operations (“MnC operations "). The memory device 222 performs or supports these operations using its multiple PIM compute elements.

[0062] The SoC 202 cooperates with the memory device 222 to perform computations across one or more bank groups of the memory device 222. The computations can be for operations or workloads that involve one or more of the processors of IP block 210. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 210, multiple IP blocks similar to the IP block 210, or both. In some implementations, the heterogeneous operation includes (or spans) a host device, e.g., HPU 214 and the memory device 222 cooperating to execute an example ML workload. In at least one example, the memory device 222 may be external to the SoC 202, whereas in another example the memory device 222 may be internal to the SoC 202.

[0063] In the example of Fig. 2, the SoC 202 is an integrated circuit of an example user / client device 230, consumer electronic device, or mobile device, where each of these devices can include items such as a smartphone 230a. tablet 230b, laptop 230c. smartwatch or wearable device 230d. The devices 230 may also include, e.g., an eNotebook, Netbook, smart speaker, or mobile computer. In some implementations, the system 200 and the SoC 202 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0064] In general, the workload task scheduler 208 determines an allocation of tasks between the multiple processors of system 200, including the HPU 214 and a processor of the memory' device 222. In some cases, the allocation of tasks is modified based on runtime operational parameters and performance metrics of one or more processors of system 200, as described below in relation to Fig. 3.

[0065] Fig. 3 illustrates an example process 300 for adjusting an allocation of tasks based on runtime conditions. A system that executes the process 300 can include a host processor 310 (e.g., the HPU 214 of Fig. 2) and a processor of a PIM block 314 (e.g., the memory device 222 of Fig. 2). In some implementations, the host processor 310 is an HPU or any ML hardware accelerator of the SoC. The two processors can execute instructions corresponding to one or more operations 302 that can be related to neural network model source code (e.g., source code 106 of Fig. 1). The example process 300 includes an evaluation of one or more performance metrics related to the execution of the operations 302 using each processor. The evaluation is performed at compile time, as described above inrelation to Fig. 1, with one or more assumptions in relation to compile-time estimations of runtime operating and performance conditions of each processor. The processing devices can be substituted with any other type and / or number of processor(s), in which each such type of processor can have different performance metrics and operational characteristics.

[0066] The host processor 310 and the processor of the PIM block 314 receive instructions from a memory controller 318 (e.g., the memory controller 205 of Fig. 2) and a physical layer 316. where the physical layer converts digital signals (i.e.. the instructions) from the memory controller 318 into physical signals, which are then transmitted over wires of one or more network fabrics 320 to the respective host processor 310 and / or the processor of the PIM block 314. In some implementations, the memory controller 318 can issue instructions to the processor of the PIM block 314 and the host processor 310. In some other implementations, the processor of the PIM block 314 and the host processor 310 dynamically request instructions. In some cases, the host processor 310 receives instructions from a host memory7311 associated with the host processor 310. In some cases, the PIM block 314 receives instructions from the host memory 311 associated with the host processor 310 via the memory controller 318. In these cases, the host processor 310 reads instructions to be executed by the processor of the PIM block 314 from the host memory 311, and transmits the instructions to the PIM block 314 via the memory' controller 318.

[0067] In some implementations, the processor of a PIM block 314 is coupled to an LPDDR DRAM 312, which is found, e.g.. in portable devices including mobile phones and tablets.

[0068] A performance estimator (e.g., the performance estimator 116 of Fig. 1) generates a set of metadata tags that represent estimates of the operational parameters for, e.g., the host processor 310 and the PIM block 314 independently, as described in relation to Figs. 1-2. In some implementations, each of the host processor metadata 334 and PIM block metadata 344 includes metadata tags of one or more of temperature, memory bandwidth, compute throughput, operating voltage, and operating frequency, corresponding to the respective processor.

[0069] The performance estimator estimates (304) performance metrics 332 and 342 (e.g., execution time, energy consumption). In some implementations, the performance estimator generates estimated performance metrics, including, e.g., execution time and energy consumption 332 and 342, based on the metadata tags for each device and the set of operations 302. For example, and as described with reference to Fig. 1, the performance estimator uses a profiling operation that uses the metadata tags and / or the operations 302, togenerate estimated performance metrics, including execution time and energy consumption for each device. As another example, and as also described with reference to Fig. 1, an analytical representation of one or more performance characteristics for a particular processor is available and can provide the estimated performance metrics 332 and 342. In both cases, e.g., the profiling and the analytical representation, the performance estimator performs a respective process that takes as an input the metadata for each processor, e.g., metadata 334 and metadata 344, along with the particular operations 302. and generates as an output the estimated performance metrics 332 and 342 (e.g., execution time, energy consumption).

[0070] In some implementations, the system that executes process 300 includes a compiler that generates (306) a host processor binary 330 and a PIM binary' 340 by compiling source code corresponding to the one or more operations 302. In some implementations, the compiler is local to the host processor 310 and / or a corresponding SoC. In some other implementations, the compiler operations are executed by a compiler on a device that is remote to the host processor 310 (and / or the SoC and the computing device on which it is implemented). Each binary’ includes executable code that corresponds to the operations 302, the respective metadata (e.g., host processor metadata 334 and PIM block metadata 344 respectively), and the estimated performance metrics (e.g., the estimated execution time and energy consumption 332 and 342 for each processor) as determined by the performance estimator.

[0071] In some implementations, the system that executes the process 300 includes a task scheduler 360 (e.g., the workload task scheduler 208 of Fig. 2) that makes (308) scheduling decisions at runtime. In some implementations, the system includes an adjustor 350, which can be implemented as logic instructions at a processor, e.g., the host processor 310. The adjustor considers runtime conditions of one or more processors (e.g., the host processor 310 and the PIM block 314), in which the runtime conditions correspond to runtime values for operational parameters, e.g., temperature, memory' bandwidth, etc.

[0072] In some implementations, the runtime conditions of the host processor 310 and the PIM block 314 are periodically monitored (e.g., after a threshold amount of time has elapsed or a threshold amount of workload tasks have been executed) as tasks are executed by the respective processor. For example, after a first allocation of tasks by7the runtime scheduler, the adjustor 350 can monitor the operational parameters of each processor, as well as observed performance metrics. Based on the runtime operational parameters and the performance metrics, the scheduler 360 can reallocate tasks between the processors to account for changing operational parameters and performance metrics.

[0073] In some cases, the metadata corresponding to runtime conditions are different from the metadata that are included with the respective binaries that correspond to the compile-time assumptions for each processor (e.g., the host processor and the PIM). As an example, the compute throughput of the host processor at runtime may be less than the anticipated runtime compute throughput as determined at compile time because of other active concurrent processes on the SoC that are competing for compute resources. As another example, the temperature of each processor may be higher or lower than the anticipated runtime temperature as determined at compile time. In some implementations, one or more hardware registers and / or counters store runtime conditions that correspond to the metadata for each processor. For example, a value of a particular hardware register can represent an average execution time for a particular processor. Other hardware registers can represent other parameters including local temperature of a particular processor, operating voltage, energy consumption, frequency, available memory bandwidth, etc. Firmware and / or software programs executed on a processor on or off of the SoC, e.g., a processor that executes the operations of the adjustor 350, can access data stored in the registers and / or counters and implement task reallocation.

[0074] In some implementations, an adjuster 350 (e.g., runtime software that is executed by a processor, e.g., a host processing unit or a CPU) can combine the compile-time metadata (i.e., the data corresponding to the estimated operational parameters) with the runtime data for these operational parameters, to obtain updated or adjusted performance metrics (e g., adjusted execution time, power performance, etc ). In some cases, observed performance and / or energy consumption of a processor that executes tasks depend on several factors. Some factors are known at compile time, and others are estimated at compile time. For example, energy consumption can depend on parameters including a number and type of executed operations, operating frequency, operating voltage, and local temperature.Although the number and type of executed operations are known at compile time, the operating frequency, voltage, and temperature can only be estimated at compile time (and these parameters may vary at runtime). At runtime, the parameter values are read from a respective hardware register / counter determine actual values of each parameter.

[0075] In some implementations, based on the adjusted performance metrics and / or the runtime operational parameters, the scheduler 360 can reallocate tasks between the host processor and the PIM. In some implementations, the scheduler 360 can access control information indicating system scheduling criteria. The system scheduling criteria can specify the target metric(s) (e.g., execution time, energy performance, performance per watt) to beevaluated for scheduling or allocating tasks to one or more processors. For example, the control information can indicate that execution time of the system should be considered in making scheduling / allocation decisions. In this example, the scheduler compares the adjusted execution time for each processor and determines that the PIM’s execution time is lower than that of the host processor. As a result, the scheduler 360 reallocates one or more tasks from the processor 310 to the PIM block 314. As another example, the control criteria can indicate that energy performance of the system should be considered in making scheduling / allocation decisions. In this example, the scheduler compares the adjusted energy performance for each processor and determines that the PIM’s energy' performance is lower than that of the host processor. As a result, the scheduler 360 reallocates one or more tasks from the processor 310 to the PIM block 314. In some implementations, the control criteria can indicate more than one criteria to consider in making scheduling decisions. In such implementations, the scheduler compares each respective performance metric specified in the control criteria for the processors and identifies the processor that has the more optimal performance across, e.g., all or a majority of the metrics. In that case, the scheduler 360 reallocates one or more tasks from the processor 310 to the PIM block 314.

[0076] In some implementations the scheduler 360 reallocates the one or more tasks based on a particular criterion or a combination of criteria. For example, the scheduler 360 can reallocate tasks based on energy efficiency, e.g., battery life, instantaneous peak current, performance, e.g., latency, or thermal throttling minimization. In addition, the schedule 360 can reallocate tasks based on a combination of two or more of the example criteria, e.g., a weighted sum of multiple criteria.

[0077] Fig. 4 is a flow diagram of an example process 400 for adjusting an allocation of tasks. Process 400 is implemented or executed by a system with at least one SoC and at least two processors. In some implementations, the two processors include a first processor of the SoC (e.g., a HPU) and a second processor external to the SoC (e.g., a PIM block). The descriptions of process 400 will reference the above-mentioned system 100, system 200, and process 300. In some examples, the steps or actions of process 400 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0078] The system generates (402) a first allocation of tasks on a SoC implemented at a device (e.g., an edge device). Examples of devices include mobile phones, tablets, laptops, smart watches, etc., as depicted by the client devices 230 in Fig. 2. In some implementations.the system generates the first allocation of tasks based on compile-time assumptions of runtime operational parameters, e.g., temperature, memory bandwidth, operating voltage, etc.

[0079] In some implementations, the first allocation of tasks relate to executing operations of layers of a neural network. In some implementations, the first allocation of tasks is executed by a HPU on the SoC.

[0080] The system identifies (404) runtime information on the device (e.g.. an edge device). In particular, the system identifies runtime information with respect to processors on the edge device, e.g., the HPU at the SoC and the processor of the PIM block, upon execution of an initial set of tasks. For example, the system can identify runtime temperature, memory bandwidth, operating voltage, etc., with respect to the HPU on the SoC and the processor of a PIM block. In some implementations, the system identifies one or more performance, environmental, and / or operational metrics after executing a subset of tasks of the first allocation of tasks. In some other implementations, the system identifies one or more environmental and / or operational metrics before executing tasks of the first allocation of tasks.

[0081] The system generates (406) a second allocation of tasks by adjusting the first allocation of tasks based on runtime information. In some implementations, the system adjusts the allocation of tasks based on the output of an adjuster, as depicted in Fig. 3. In some cases, the realized runtime information is different from the presumed runtime information determined at compile time used to generate the first allocation of tasks. In these cases, a different allocation of tasks between one or more processors of the system may be more energy efficient or provide better performance metrics. For example, the first allocation of tasks can allocate all tasks to the HPU at the SoC based on a comparison of performance metrics between the HPU and the PIM processor at compile time. At runtime, the system determines an updated comparison of performance metrics, and may reallocate one or more of the tasks to the PIM based on the realized set of runtime parameters, e.g., temperature, memory bandwidth, etc.

[0082] The system executes (408) a task provided by the second allocation of tasks on a PIM block of an integrated memory device coupled to the SoC. In the case in which the runtime conditions indicate improved performance metrics by allocating a task to the PIM instead of the SoC, the system can adjust the allocation of tasks accordingly.

[0083] Fig. 5 is a flow diagram of an example process 500 for adjusting an allocation of tasks. Process 500 is implemented or executed by a system with at least one SoC and at least two processors (e.g., a HPU and a processor of a PIM block). The descriptions ofprocess 500 will reference the above-mentioned system 100, system 200, and process 300. In some examples, the steps or actions of process 500 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non- transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0084] The system obtains (502) a first task allocation that allocates a first set of tasks between a first processor of the SoC and a second processor external to the SoC. In some implementations, (1) the first processor is a machine-learning (ML) hardware accelerator of the SoC, which as explained with reference to Figs. 1 and 2, can include the HPU, and (2) the second processor external to the SoC can include a processing unit of an integrated memory device coupled to the SoC (such as, e.g., a PIM block described above with reference to Figs. 1 and 2). In some implementations, a scheduler (e.g., the workload task scheduler 208 of Fig. 2) allocates the first set of tasks between the first and second processors.

[0085] The system identifies (504) runtime information determined upon execution of a subset of the first set of tasks. As explained with reference to Figs. 1-3, the runtime information can include data for operational parameters related to a particular processor, including its temperature, memory bandwidth, compute throughput, etc. In some implementations, the runtime information includes (1) a first set of runtime parameter values (e.g., temperature, memory bandwidth, and / or one or more other operational parameters described above with reference to Figs. 1-3) that are generated upon the first processor executing tasks allocated according to the first task allocation and (2) a second set of runtime parameter values (e.g., temperature, memory bandwidth, and / or one or more other operational parameters described above with reference to Figs. 1-3) that are generated upon the second processor executing tasks allocated according to the first task allocation. In some implementations, an adjustor, implemented as logic defined in hardware of the SoC or in software executed by a CPU or HPU of the SoC, obtains the runtime information upon execution of the subset of the first set of tasks. In some cases, the adjustor accesses stored values of one or more registers and / or counters with particular pre-defined addresses, e.g., read address 0x1000 represents a value for a current compute clock frequency of a particular processor. The adjustor reads the value stored at the pre-defined addresses to determine runtime parameters to adjust a performance and / or energy’ consumption evaluation.

[0086]

[0087] As will be appreciated by one skilled in the art, in some instances, the runtime information reveals values for operational parameters that may be different from theestimated operational parameters for executing a workload, as estimated at compile time (e.g., using the analytical modeling or profiling techniques described above). As noted above with reference to Figs. 1-3, a set of estimated operational parameters are obtained for each of the first processor and the second processor at compile time, and a set of estimated performance metrics are computed for each processor (e.g., by the performance estimator) using the estimated operational parameters for the respective processor. Based on these estimated operational parameters and performance metrics, a scheduler (e.g.. the workload task scheduler 208 of Fig. 2) allocates tasks for a workload (e.g., a machine learning workload) between the first processor and the second processor.

[0088] For the remaining tasks in the first set of tasks, the system adjusts (506), the first task allocation based on the runtime parameters to obtain a second task allocation, in which the second task allocation may result in the remaining tasks of the workload being allocated differently between the first processor and the second processor relative to the first task allocation. In some implementations, the adjustor, executed as software or hardware logic of the SoC. combines the operational parameter estimates with the runtime information associated with these parameters, to generate adjusted performance metrics (e.g.. adjusted execution time, adjusted energy performance, etc.).

[0089] It may be the case in some instances that a first processor, e.g., a processor of a PIM block, executes a particular operation ith lower latency and / or lower energy consumption compared to a second processor, e.g., a host processor of the SoC executing the same particular operation. In some cases, a difference in performance and / or energy consumption can depend on the type of operation. For example, operations of a particular type of neural network layer can be executed with lower latency on the first processor compared to the second processor. In some case, the comparison between the first and the second processor can depend on operational and / or environmental parameters like operating voltage and operating frequency. For example, the first processor can execute a particular operation with lower latency when it operates with a first operating voltage and operating frequency, but with a higher latency than the second processor with a second operating voltage and operating frequency. For this reason, the system determines the runtime operating voltage and operating frequency to determine an optimal processor to execute the particular operation.

[0090] In some implementations, the scheduler, (e.g., the scheduler 360 of Fig. 3) can access control information indicating system scheduling criteria. The system scheduling criteria can specify a target metric or target metrics (e.g., execution time, energyperformance, performance per wat) for scheduling or allocation of tasks to one or more processors. Using the control information specifying scheduling criteria / target metrics, the scheduler can identify the respective adjusted performance metric of the two processors and compare their values (as described above with reference to Fig. 3). As part of this comparison, the scheduler may determine that the PIM block has beter energy performance and / or execution time relative to the host processor, in which case, the scheduler can reallocate the remaining tasks differently between the host processor and the PIM block, with additional tasks being reallocated to the PIM block. The scheduler, or another process executed by a processor can initiate, based on an output of the scheduler, a command for a particular task or set of tasks to the host processor or the PIM block.

[0091] The system executes (508) one or more tasks in the remaining tasks based on the second task allocation. In some cases, if the first task allocation allocated all tasks to the first processor, the second task allocation may allocate all or some of remaining tasks of a workload (after executing an initial set of tasks of the workload) to the second processor. The second processor and / or the first processor then execute the remaining tasks in the workload based on the second task allocation — which as explained above and throughout this document is determined dynamically based on runtime conditions.

[0092] Embodiments of the subject mater and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[0093] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0094] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purposelogic circuitry', e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0095] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a scnpt. or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0096] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication netw ork.

[0097] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry', e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0098] Computers suitable for the execution of a computer program, by way of example, can be based on general or special purpose microprocessors or both, or any' other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.

[0099] Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone.a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g.. a universal serial bus (USB) flash drive, to name just a few.

[0100] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory' devices, including by way of example semiconductor memory devices, e.g.. EPROM. EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0101] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0102] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data serv er, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network (“WAN’’), e.g., the Internet.

[0103] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0104] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0105] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.What is claimed is:

Claims

CLAIMS1. A method performed using a System-on-Chip (“SoC”), the method comprising: obtaining a first task allocation that allocates a first set of tasks between a first processor of the SoC and a second processor external to the SoC; identifying runtime information determined upon execution of a subset of the first set of tasks; adjusting the first task allocation based on the runtime information to obtain a second task allocation that allocates remaining tasks in the first set of tasks; and executing one or more tasks based on the second task allocation.

2. The method of claim 1, wherein: i) the first processor is a machine-learning (ML) hardware accelerator of the SoC; and ii) the second processor is a processing unit in an integrated memory device coupled to the SoC.

3. The method of claim 1, further comprising: obtaining, at compile time, a set of estimated operational parameters for each of the first processor and the second processor; and computing, for each of the first processor and the second processor, a set of estimated performance metrics using the estimated operational parameters for the respective processor, wherein the first task allocation is based on the sets of estimated operational parameters and the sets of estimated performance metrics.

4. The method of claim 3, wherein the runtime information comprises: a first set of runtime parameter values that are generated upon the first processor executing tasks allocated according to the first task allocation; and a second set of runtime parameter values that are generated upon the second processor executing tasks allocated according to the first task allocation.

5. The method of claim 4, wherein adjusting the first task allocation based on the runtime information comprises:adjusting a set of estimated performance metrics for the first processor using the first set of runtime parameter values and the set of estimated performance metrics for the first processor, to obtain a set of adjusted performance metrics for the first processor; adjusting a set of estimated performance metrics for the second processor using the second set of runtime parameter values and the set of estimated performance metrics for the second processor, to obtain a set of adjusted performance metrics for the second processor; identifying, using control information, a particular adjusted performance metric from each set of adjusted performance metrics; comparing the identified particular adjusted performance metrics; and based on the comparison of the identified adjusted performance metrics, adjusting the first task allocation to obtain the second task allocation.

6. The method of claim 5, wherein the control information specifies a target metric to be evaluated for allocating tasks between the first processor and the second processor.

7. The method of claim 5, wherein the second task allocation allocates the remaining tasks differently between the first processor and the second processor relative to the first task allocation.

8. The method of claim 7, wherein: the first task allocation allocates the first set of tasks to the first processor, and the second task allocation allocates a first subset of the remaining tasks to the first processor and a second subset of the remaining tasks to the second processor.

9. The method of claim 8, wherein executing one or more tasks in the remaining tasks based on the second task allocation comprises: executing, by the first processor and according to the second task allocation, the first subset of the remaining tasks; and executing, by the second processor and according to the second task allocation, the second subset of the remaining tasks.

10. The method of claim 1, wherein each of the SoC and the second processor is implemented at a same edge device.

11. A system comprising: one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: obtaining a first task allocation that allocates a first set of tasks between a first processor of the SoC and a second processor external to the SoC; identifying runtime information determined upon execution of a subset of the first set of tasks; adjusting the first task allocation based on the runtime information to obtain a second task allocation that allocates remaining tasks in the first set of tasks; and executing one or more tasks based on the second task allocation.

12. The system of claim 11, wherein: i) the first processor is a machine-learning (ML) hardware accelerator of the SoC; and ii) the second processor is a processing unit in an integrated memory device coupled to the SoC.

13. The system of claim 1, wherein the operations further comprising: obtaining, at compile time, a set of estimated operational parameters for each of the first processor and the second processor; and computing, for each of the first processor and the second processor, a set of estimated performance metrics using the estimated operational parameters for the respective processor, wherein the first task allocation is based on the sets of estimated operational parameters and the sets of estimated performance metrics.

14. The system of claim 13, wherein the runtime information comprises: a first set of runtime parameter values that are generated upon the first processor executing tasks allocated according to the first task allocation; and a second set of runtime parameter values that are generated upon the second processor executing tasks allocated according to the first task allocation.

15. The system of claim 14, wherein adjusting the first task allocation based on the runtime information comprises:adjusting a set of estimated performance metrics for the first processor using the first set of runtime parameter values and the set of estimated performance metrics for the first processor, to obtain a set of adjusted performance metrics for the first processor; adjusting a set of estimated performance metrics for the second processor using the second set of runtime parameter values and the set of estimated performance metrics for the second processor, to obtain a set of adjusted performance metrics for the second processor; identifying, using control information, a particular adjusted performance metric from each set of adjusted performance metrics; comparing the identified particular adjusted performance metrics; and based on the comparison of the identified adjusted performance metrics, adjusting the first task allocation to obtain the second task allocation.

16. The method of claim 15, wherein the control information specifies a target metric to be evaluated for allocating tasks between the first processor and the second processor.

17. The method of claim 5, wherein the second task allocation allocates the remaining tasks differently between the first processor and the second processor relative to the first task allocation.

18. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining a first task allocation that allocates a first set of tasks between a first processor of the SoC and a second processor external to the SoC; identifying runtime information determined upon execution of a subset of the first set of tasks; adjusting the first task allocation based on the runtime information to obtain a second task allocation that allocates remaining tasks in the first set of tasks; and executing one or more tasks based on the second task allocation.

19. The one or more computer-readable storage media of claim 18, wherein the operations further comprising: obtaining, at compile time, a set of estimated operational parameters for each of the first processor and the second processor; andcomputing, for each of the first processor and the second processor, a set of estimated performance metrics using the estimated operational parameters for the respective processor, wherein the first task allocation is based on the sets of estimated operational parameters and the sets of estimated performance metrics.

20. The one or more computer-readable storage media of claim 19, wherein the runtime information comprises: a first set of runtime parameter values that are generated upon the first processor executing tasks allocated according to the first task allocation, and a second set of runtime parameter values that are generated upon the second processor executing tasks allocated according to the first task allocation; and wherein adjusting the first task allocation based on the runtime information comprises: adjusting a set of estimated performance metrics for the first processor using the first set of runtime parameter values and the set of estimated performance metrics for the first processor, to obtain a set of adjusted performance metrics for the first processor; adjusting a set of estimated performance metrics for the second processor using the second set of runtime parameter values and the set of estimated performance metrics for the second processor, to obtain a set of adjusted performance metrics for the second processor; identifying, using control information, a particular adjusted performance metric from each set of adjusted performance metrics; comparing the identified particular adjusted performance metrics; and based on the comparison of the identified adjusted performance metrics, adjusting the first task allocation to obtain the second task allocation.

Images