ADAPTIVE SCHEDULING FOR EXECUTING MACHINE LEARNING OPERATIONS WITHIN A MULTIPROCESSOR COMPUTING DEVICE - Patent application

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for scheduling execution of machine learning model operations on a multiprocessor computing device. The method generally includes measuring a temperature for each of a plurality of locations on the computing device during execution of operations in a first portion of the machine learning model on a first processing unit of the computing device. The measured temperature for the first processing unit is determined to exceed a threshold temperature. A second processing unit of the computing device is selected to be used in executing operations in a second portion of the machine learning model based on one or more operating parameters for the computing device. Execution of the operations in the second portion of the machine learning model on the second processing unit is scheduled.

Description

[Technical field]

[0001] introduction Aspects of the present disclosure relate to machine learning, and more particularly, to scheduling execution of portions of a machine learning model on a computing device having multiple processors.

[0002]

[0002] In various computing systems, such as on smartphones, tablet computers, or the like, multiple processors may be used to perform various computational tasks, such as processing different parts of a neural network (e.g., different layers, groups of layers, network branches, sub-networks, etc.) or other machine learning models. These processors may have different performance, power usage, and thermal characteristics. For example, in a computing device that includes one or more central processing unit (CPU) cores, one or more graphics processing units (GPUs), neural processing units (NPUs), one or more digital signal processors (DSPs), and the like, each of these processors may be suited (e.g., specifically optimized) for a particular type of task and may generate different amounts of heat when loaded. For example, a CPU core may perform less well for a particular task than a more specialized processing unit, such as a GPU, NPU, or DSP, and may use more power and therefore generate more heat when loaded. Conversely, a more specialized processing unit may provide better performance while consuming less power and generating less heat than a specialized processing unit performing the same task.

[0003]

[0003] Processing units within a computing device may generally operate within a thermal window defined by lower and upper operating temperatures for those processing units, as well as for other components within the computing device (e.g., case temperature for mobile devices so that the computing device can be held by a user without burning the user, battery temperature to minimize the possibility of thermal runaway or other adverse effects of heat on battery life, etc.). As the processing units approach various threshold temperatures up to the thermal upper limit, various actions may be taken to reduce the amount of heat generated by those processing units. For example, the core voltage for the processor may be reduced, reducing the clock speed at which the processing units operate. Reducing the clock speed at which the processing units operate may result in less heat being generated, but doing so may also reduce system performance.

[0004]

[0004] Therefore, what is needed are improved techniques for thermal management in computing systems that perform machine learning operations. Summary of the Invention

[0005]

[0005] Certain aspects provide a computer-implemented method for scheduling execution of machine learning model operations on a multiprocessor computing device. The method generally includes measuring a temperature for each of a plurality of locations on the computing device during execution of operations in a first portion of the machine learning model on a first processing unit of the computing device. The measured temperature for the first processing unit is determined to exceed a threshold temperature. A second processing unit of the computing device is selected to be used in executing operations in a second portion of the machine learning model based on one or more operating parameters for the computing device. Execution of the operations in the second portion of the machine learning model on the second processing unit is scheduled.

[0006]

[0006] Another aspect provides a processing system configured to perform the aforementioned methods as well as methods described herein; a non-transitory computer readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods as well as methods described herein; a computer program product embodied on the computer readable storage medium including code for performing the aforementioned methods as well as methods further described herein; and the processing system comprising means for performing the aforementioned methods as well as methods further described herein.

[0007] The following description and the annexed drawings set forth in detail certain illustrative features of the one or more embodiments. [Brief description of the drawings]

[0008]

[0008] The accompanying drawings illustrate several aspects of one or more embodiments and therefore should not be considered as limiting the scope of the present disclosure. [Figure 1]

[0009] 1 illustrates an example layout of a multi-processor computing device. [Diagram 2]

[0010] 1 illustrates an example architecture of a neural network in which layers in the neural network are organized into groups configured for processing on different sets of processing units in a computing device, according to aspects of the present disclosure. [Diagram 3]

[0011] 1 illustrates example operations for adaptively scheduling execution of operations in a machine learning model on a multi-processor computing device in accordance with an aspect of the present disclosure. [Figure 4]

[0012] 1 illustrates an example implementation of a processing system in which adaptive scheduling of execution of operations in a neural network may be performed, in accordance with aspects of the present disclosure.

[0009]

[0013] For ease of understanding, wherever possible, the same reference numbers have been used to designate identical elements common to the figures. It is contemplated that elements and features of one embodiment may be advantageously incorporated in other embodiments without further recitation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010]

[0014] Aspects of the present disclosure provide techniques for adaptively scheduling the execution of operations in a neural network on a multi-processor computing device.

[0011]

[0015] Machine learning models such as convolutional neural networks, recurrent neural networks, and the like can be used for various tasks. For example, neural networks can be used for spatial scaling using artificial intelligence techniques in adjusting the resolution of an image (e.g., increasing the resolution of an input image using super-resolution techniques), adjusting the appearance of an image (e.g., through image fusion, applying various color effects such as introducing background or foreground blur (also known as "bokeh"), etc., for generating high dynamic range (HDR) images, temporal interpolation to allow frames to be generated at a higher frame rate (e.g., corresponding to the refresh rate of the display on which the frames will be displayed). In various scenarios such as video playback, gameplay, or the like, these operations can be performed in real time or near real time, and therefore a significant amount of processing power can be allocated to performing these tasks.

[0012]

[0016] Performing these various computational tasks in real-time or near real-time using neural networks can be computationally expensive tasks, so the processors on which these tasks are executed may draw a significant amount of power in order to allocate a sufficient amount of computational resources to these tasks. However, due to the resistive properties of various components and circuits within a computing device, an increase in current draw may result in a corresponding increase in operating temperature for various components of the computing device. In some cases, as the computing device continues to perform operations using the neural network, components within the computing device may reach one or more temperature limits defined for safe and reliable operation of the computing device (e.g., a maximum temperature defined for the processor before the processor is damaged, a maximum temperature defined for the battery to avoid thermal runaway or other destructive events, a maximum temperature defined for the case of the computing device to prevent the user from being burned, etc.). Upon reaching these temperature limits, various actions may be taken to keep the computing device within its defined temperature limits. For example, the clock speed at which these processors operate and the corresponding current draw for these processors may be reduced. However, reducing the clock speed may increase the amount of time required for the operations to be performed. For tasks that must be performed in real-time or near real-time, such as video processing, in-game image rendering, and the like, this can introduce "choppiness" into the computations. For example, in video processing applications, frames may be dropped, which can result in the resulting video not looking smooth.

[0013] Exemplary Computing Device Architecture

[0017] 1 illustrates an exemplary computing device 100 including multiple processors on which operations in a neural network may be performed. As illustrated, computing device 100 includes multiple central processing unit (CPU) cores 110, multiple central processing unit (CPU) cores 120, a graphics processing unit (GPU) 130, multiple digital signal processors (DSPs) 140, a neural processing unit (NPU) 150, memory 160 accessible by CPU cores 110, CPU cores 120, GPU 130, DSP 140, and NPU 150, and multiple temperature sensors 170.

[0014]

[0018] The multiple CPU cores 110 and the multiple CPU cores 120 may, in some aspects, be part of a heterogeneous CPU architecture (such as the big.LITTLE architecture in an Advanced RISC Machine (ARM) processor) in which the CPU core 110 is designated as a “performance” core and the CPU core 120 is designated as an “efficiency” core. The CPU core 110 may provide additional processing power to the CPU core 120, but may draw more power to provide this additional processing power. In contrast, the CPU core 120 may be a power-efficient core that allows for the completion of computational tasks with lower computational complexity using less power than the CPU core 110. Therefore, various operating parameters may be used to schedule the execution of tasks across the CPU cores 110 and 120 to leverage the processing power of the CPU core 110 and the power efficiency of the CPU core 120. For example, complex computational tasks, such as tasks involving large numbers or floating-point data types, may be scheduled for execution on CPU core 110, while tasks involving small numbers or integer data types may be scheduled for execution on CPU core 120 because these tasks may not require leveraging the additional processing characteristics of CPU core 110.

[0015]

[0019] GPU 130 may generally be a processing unit including multiple units that may enable parallel execution of various operations. In some aspects, GPU 130 may be configured to perform vector math operations and other complex math operations that may be used to generate graphics for display on a display device connected to or integral with computing device 110. Because GPU 130 may support parallel processing for complex tasks such as vector math operations, GPU 130 may also be used in performing various tasks in neural networks with a higher performance level than that of CPU core 110 or CPU core 120.

[0016]

[0020] DSP 140 may perform signal processing on various data inputs to generate processed signal outputs. In some aspects, these DSPs may be associated with various image capture, audio capture, or other data capture devices connected to or integral with computing device 100. For example, DSP 140 A may be associated with a first camera (e.g., a wide-angle camera) of the mobile device, and the DSP 140 B may be associated with a second camera of the mobile device (e.g., a camera with a “normal” field of view, or a field of view closer to that of a human), and the DSP 140 C may be associated with a video camera of a mobile device, and so on.

[0017]

[0021] NPU 150 may generally be a specialized processing unit that performs operations related to various machine learning algorithms. In general, NPU 150 may perform various prediction tasks, convolution tasks, subsampling tasks, and the like within a neural network. For example, NPU 150 may be designed to process data structured as high-dimensional tensors in parallel and may include buffers of sufficient size to enable data reuse within a neural network. Because NPU 150 is a specialized processing unit tailored to perform tasks within a neural network, general tasks that may be performed on CPU core 110 and / or CPU core 120 may not be scheduled on NPU 150.

[0018]

[0022] During execution of various operations on computing device 100, scheduler 180 may periodically or aperiodically measure the operating temperatures for each of CPU core 110, CPU core 120, GPU 130, DSP 140, and NPU 150 via temperature sensors 170. As shown, a varying number of temperature sensors may be implemented on each processing unit within computing device 100. For example, each of CPU cores 110 (e.g., a “high performance” core) may include two temperature sensors 170, while each of CPU cores 120 (e.g., a “high efficiency” core) may include a single temperature sensor 170. Although not shown, additional temperature sensors 170 may also be implemented within computing device 100 to measure the operating temperatures of other components of the computing device.

[0019]

[0023] As described above, each of the processing units (and other components) in computing device 100 may have a specified upper operating temperature limit. Generally, a processing unit may not be allowed to operate above the specified upper operating temperature limit for the processing unit because exceeding the specified upper operating temperature limit may cause damage to the processing unit or to other components in computing device 100. Typically, to prevent a processing unit from exceeding the specified upper operating temperature limit, various power control techniques may be used to reduce the clock speed (or rate at which instructions are executed) of the processing unit, thus reducing the amount of heat generated by the processing unit. However, reducing the clock speed of a processing unit may adversely affect operations being performed on the processing unit.

[0020]

[0024] To allow machine learning operations (e.g., via a neural network) to be adaptively performed on computing device 100, temperature measurements, known distances between different processing units on computing device 100, and other operating parameters may be used to schedule the execution of the machine learning operations. In general, because the layout of processing units in computing device 100 is known and fixed for the specific design of a particular computing device 100, and because heat may be assumed to dissipate as a function of distance, scheduler 180 may assume that a processing unit that is farther away from a processing unit currently performing a neural network operation may be more suitable for use in future execution of the neural network operation than a processing unit that is closer to the processing unit currently performing the machine learning operation. In effect, scheduler 180 may therefore move execution to a processing unit that is cooler than the current processing unit currently performing the machine learning operation, while maintaining the processing power required to perform the machine learning operation with a desired performance level (e.g., generating frames in a video or streaming content that follows a specified refresh rate such that the transition between frames appears smooth).

[0021]

[0025] For example, assume that a machine learning operation may be configured for execution on NPU 150, GPU 130, and CPU cores 110, in that order of priority. Thus, assuming that each of NPU 150, GPU 130, and CPU cores 110 are idle and below a thermal threshold (e.g., a thermal cap), the machine learning operation may be scheduled for execution initially by NPU 150. R ~170 T If one or more of the thermal thresholds indicate that the temperature of NPU 150 is approaching or at the thermal threshold, scheduler 180 may determine that the subsequent execution of the machine learning operation must be moved to GPU 130 or one or more CPU cores 110. At this point, assuming that GPU 130 and CPU core 110 have measured temperatures below the thermal upper limit defined for these processing units, scheduler 180 may consider the distance from NPU 150 and other operating parameters in identifying whether GPU 130 or CPU core 110 will be used in the subsequent execution of the machine learning operation. In this example, because GPU 130 is closer to NPU 150 than CPU core 110, scheduler 180 may consider GPU 130 a less suitable candidate for use in the subsequent execution of the machine learning operation than CPU core 110. Therefore, scheduler 180 may schedule the subsequent execution of the machine learning operation for one or more of CPU cores 110. In some aspects, the distance metric may be calculated for each processing unit. In such a case, the CPU core 110 A However, since it is the processing unit furthest from the NPU 150, the CPU core 110 A may be selected as the processing unit for use during subsequent execution of the machine learning operation on the computing device 100.

[0022]

[0026] Other operating parameters may alternatively or additionally be used to determine which processing units will be used in the subsequent execution of the machine learning operation on the computing device 100. For example, the current load on each processing unit may be considered in determining whether a particular processing unit is a candidate for use in the subsequent execution of the machine learning operation. If the processing unit has a current load that exceeds a threshold, the processing unit may not be considered a suitable candidate for the subsequent execution of the machine learning operation. This threshold may be defined a priori or may be configured for each processing unit, for example, based on the performance characteristics of each processing unit. A more powerful processing unit (e.g., a processing unit that can process a larger number of operations over a given period of time) may have a higher usage threshold than a less powerful processing unit, for example. This is because a more powerful processing unit may have additional resources that may be dedicated for use in the execution of other operations than a less powerful processing unit. In another example, the current temperature of each processing unit may be used in determining whether the processing unit is a candidate for the subsequent execution of the machine learning operation. In general, because machine learning operations will be computationally intensive and therefore likely to increase the temperature of any processing unit on which they are performed, processing units having temperatures closer to their thermal upper limit or other prescribed thermal thresholds may be less suitable for use in subsequent performance of machine learning operations than processing units having temperatures farther away from their thermal upper limit or other thermal thresholds. However, it should be understood that current load, temperature, and distance from a processing unit currently performing a machine learning operation are only examples of operational parameters that may be considered in selecting a processing unit on which a subsequent machine learning operation will be performed, and that other operational parameters may be used in conjunction with or in place of the operational parameters described herein.

[0023] Exemplary Neural Network Architecture

[0027] FIG. 2 illustrates an example architecture of a neural network 200 in which layers in the neural network are organized into groups configured for processing on different sets of processing units in a computing device (e.g., one or more of CPU core 110, CPU core 120, GPU 130, DSP 140, and NPU 150 of computing device 100 shown in FIG. 1 ), in accordance with aspects of the present disclosure.

[0024]

[0028] In general, neural network 200 may include multiple layers, each of which may have a different level of complexity. For example, layers closer to the inputs used to generate feature maps from the inputs may be the most complex layers in the neural network, while layers performing operations on subsampled data sets output from previous layers may be simpler. Because the complexity involved in performing operations in a neural network is known or can at least be estimated (e.g., based on the number of parameters in the neural network, the number of nodes, the input size, etc.) to decrease the closer the neural network is to generating outputs that represent the inputs, layers in the neural network may be grouped into multiple groups. Each group may be configured for execution on a particular processing unit in computing device 100, and priorities for performing operations within groups of layers in the neural network may also be defined.

[0025]

[0029] For example, as shown, neural network 200 may be grouped into a first group 210, a second group 220, and a third group 230. The first group 210 of layers in neural network 200 may be configured for execution on DSP 140, NPU 150, and CPU cores 110 and / or 120, in that order of preference. The second group 220 of layers in neural network 200, which may be considered to have a lower computational complexity than the layers in first group 210, may be configured for execution on DSP 140, NPU 150, and GPU 130, in that order of preference. Finally, the third group 230 of layers, which may be considered to have a lower computational complexity than the layers in second group 220, may be configured for execution on CPU cores 110 and / or 120, and GPU 130, in that order of preference.

[0026]

[0030] In some aspects, different layers in the neural network 200 may be configured for execution with different quantization levels, or different granularities at which data may be generated. To allow different groups of layers to be configured, each layer included within a group of layers may be trained or compiled for execution with the same quantization levels. For example, a first group of layers 210 (which may correspond to a group of layers having the highest computational complexity, as discussed above) may be trained or compiled for execution with a 32-bit floating-point number space (e.g., −3.4 * 10 38 ~3.4 * 10 38The first group of layers 210 may be trained or compiled for quantization within a space that spans 0 to 255 (e.g., 128 to 127 if signed, or 0 to 255 if unsigned). The second group of layers 220 (which may correspond to a group of layers with lower computational complexity than the first group of layers 210) may be trained or compiled for execution with a lower computational complexity data type, such as quantization within an 8-bit integer space (e.g., a space that spans −128 to 127 if signed, or 0 to 255 if unsigned). The third group of layers 230 may then be trained or compiled for execution with an even lower computational complexity data type, such as quantization within a smaller integer space (e.g., 6 bits, 4 bits, etc.). In general, these different quantization levels may be constraints that limit the selection of processors to which subsequent machine learning operations may be transferred and / or may make some processors better targets for scheduling subsequent machine learning operations. For example, if layers in a neural network are compiled for floating-point quantization, then processors with greater floating-point processing capabilities (e.g., “high performance” CPU cores, GPUs, NPUs, etc.) may be suitable for performing operations in those layers, while processors that lack floating-point processing capabilities or have limited floating-point processing capabilities (e.g., “high efficiency” CPU cores, etc.) may not be suitable for performing operations in those layers.

[0027]

[0031] Although FIG. 2 illustrates a division of a neural network into multiple groups of layers, it should be recognized that any machine learning model may be organized into similar groupings, and that the division of a neural network into multiple groups of layers is merely one example of a division of a machine learning model into different groups of components that may be scheduled for execution on different groups of processing units.

[0028] Exemplary Method for Adaptive Scheduling of Machine Learning Operations in a Multiprocessor Computing Device

[0032] FIG. 3 illustrates example operations 300 that may be performed to adaptively schedule execution of operations in a machine learning model on a multi-processor computing device, such as the computing device 100 shown in FIG. 1, in accordance with aspects of the disclosure.

[0029]

[0033] As shown, the operation 300 may begin at block 310. During execution of operations within a first portion of the machine learning model of a first processor of a multiprocessor computing device, a temperature for each processing unit of a plurality of processing units is measured. The measurements may be obtained by interrogating or otherwise polling one or more temperature sensors (e.g., temperature sensor 170 shown in FIG. 1), each sensor associated with a particular processing unit or other individual component within the computing device. In some aspects where a processing unit has multiple temperature sensors associated therewith, various techniques may be used to determine the temperature of the processing unit. For example, an average temperature across the multiple temperature sensors may be used as the temperature of the processing unit. In another example, the highest temperature measured across the multiple temperature sensors may be used as the temperature of the processing unit. In some aspects, the measured temperature may correspond to an instantaneous temperature reading, while in other aspects, the measured temperature may be a running average over the most recent multiple samples.

[0030]

[0034] At block 320, the operation 300 proceeds with determining that the measured temperature for the first processing unit exceeds a threshold temperature.

[0031]

[0035] At block 330, the operations 300 proceed with selecting a second processing unit of the computing device for use in executing the operations in the second portion of the machine learning model. Generally, the second processing unit may be selected based on one or more operating parameters for the computing device. These operating parameters may include, for example, the distance between the first processing unit and the second processing unit, the current load on the second processing unit, the current temperature of the second processing unit, or the like. Information such as the distance between the first processing unit and the second processing unit may be specified, for example, in a configuration file. For example, the configuration file may include information corresponding to the physical layout of the computing device on which the machine learning operations are performed and identifying the location of each processing unit in the computing device. In some aspects, the configuration file may include information identifying the particular slots in which these processors are installed, and various assumptions may be made based on this information. For example, in a computing system in which processors are installed in expansion slots numbered 0 to n, with expansion slot 0 closest to the CPU, the relative distance between each processor in the computing system may be determined and used in selecting the processor to be used in executing the operations in the second portion of the machine learning model.

[0032]

[0036] In some aspects where distance from the first processing unit is one of the operational parameters used to select the second processing unit, the second processing unit may be selected as the processing unit that is farthest from the first processing unit and has a temperature below a threshold temperature (e.g., a thermal upper limit defined for the processing unit, one of multiple thermal thresholds that define different performance levels for the processing unit, etc.).

[0033]

[0037] In some aspects, the second processing unit may be further selected based on an order of types of processing units to be used in performing operations in the second portion of the neural network. These orders may be based, for example, on the size of the data to be processed in the second portion of the machine learning model (e.g., the data types used to process the data in the second portion of the machine learning model, such as 32-bit floating point, 8-bit integer, etc.) and the performance levels associated with each type of processing unit in the computing device (e.g., the number of floating point operations per second, integer operations per second, etc. supported by the processing unit of the computing device).

[0034]

[0038] In some aspects, the second processing unit may be selected by identifying a set of processing units having a distance from the first processing unit that exceeds a distance threshold and a measured temperature that is below a threshold temperature. The distance threshold may be an absolute distance between the processing units (e.g., according to a known architectural layout of the computing device) or an assumed distance based on general rules that govern the location of processing units and expansion slots within the computing device. The second processing unit may be selected from the identified set of processing units. For example, the second processing unit may be selected from the identified set of processing units according to an ordering of these processing units (as described above), a current load on these processing units, etc. In some aspects, the identified set of processing units may be further identified by identifying processing units that have a current load that is less than a threshold load.

[0035]

[0039] At block 340, execution of operations in a second portion of the machine learning model is scheduled for execution on a second processing unit.

[0036]

[0040] In some aspects, the first portion of the machine learning model and the second portion of the machine learning model may be different layers of a neural network configured for execution on the same set of processing units. For example, the first portion of the machine learning model and the second portion of the machine learning model network may both be layers of a neural network configured to execute on the same type of processor using the same data types.

[0037]

[0041] In some aspects, the first portion of the machine learning model may be a layer in a first set of layers of a neural network, and the second portion of the machine learning model may be a layer in a second set of layers of the neural network. The first set of layers may be configured for execution on a first set of processing units of a computing device. The second set of layers may be configured for execution on a second set of processing units of a computing device. For example, as described above with respect to FIG. 2, the first set of layers of the neural network and the second set of layers of the neural network may include sets of layers having different computational complexity levels. In general, computational complexity information, such as data types used by different parts of the machine learning model (and corresponding assumptions regarding the computational complexity of operations in these different parts of the machine learning model), may be known in advance (e.g., through configuration information accompanying the machine learning model) and may be used to schedule subsequent machine learning operations on the computing device. The first set of layers may be configured with a first set of quantization parameters, and the second set of layers may be configured with a second set of quantization parameters. The second set of quantization parameters may, for example, correspond to quantization over a smaller data type than the first set of quantization parameters. For example, the first set of layers may include layers within a neural network configured to process data using 32-bit floating point numbers, while the second set of layers may include layers within a neural network configured to process data using 8-bit integers. Of course, it should be recognized that this is merely one example and that the first and second sets of layers may be configured to process and quantize data using a variety of data types.

[0038]

[0042] In some aspects, the first set of processing units may include a neural processing unit (NPU), a digital signal processor (DSP), and multiple central processing unit (CPU) cores, while the second set of processing units may include multiple CPU cores and multiple graphics processing unit (GPU) processors. In such a case, if the first portion of the neural network is executing on one or more CPU cores, then the second portion of the neural network may be scheduled for execution using the GPU processors because the CPU cores are relatively closer to the CPU cores on which the first portion of the neural network executes, and therefore it may be assumed that the CPU may not be a suitable candidate processing unit for use in performing operations with the second set.

[0039] Exemplary Processing System for Adaptive Scheduling of Machine Learning Operations in a Multiprocessor Computing Device

[0043] FIG. 4 illustrates an example processing system 400 for adaptively scheduling the execution of operations in a neural network on a multi-processor computing device, such as that described herein with respect to FIG. 3.

[0040]

[0044] The processing system 400 includes a central processing unit (CPU) 402, which in some examples may be a multi-core CPU. Instructions executed in the CPU 402 may be loaded, for example, from a program memory associated with the CPU 402 or from a partition in memory 424.

[0041]

[0045] The processing system 400 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 404, a digital signal processor (DSP) 406, a neural processing unit (NPU) 408, a multimedia processing unit 410, and wireless connectivity components 412.

[0042]

[0046] An NPU, such as 408, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic to execute machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), etc. An NPU may alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graph processing unit.

[0043]

[0047] An NPU, such as 408, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-chip (SoC), while in other examples, they may be part of a dedicated neural network accelerator.

[0044]

[0048] NPUs can be optimized for training or inference, or in some cases can be configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0045]

[0049] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, to improve model performance. In general, optimization based on mispredictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0046]

[0050] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs may be configured to input new data and rapidly process the data through already trained models to generate model outputs (e.g., inferences).

[0047]

[0051] In one implementation, the NPU 408 is part of one or more of the CPU 402, the GPU 404, and / or the DSP 406.

[0048]

[0052] In some examples, the wireless connectivity component 412 may include sub-components for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 412 is further connected to one or more antennas 414.

[0049]

[0053] The processing system 400 may also include one or more sensor processing units 416 associated with any type of sensor, such as the temperature sensor 170 shown in FIG. 1 , one or more image signal processors (ISPs) 418 associated with any type of image sensor, and / or a navigation processor 420, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0050]

[0054] Processing system 400 may also include one or more input and / or output devices 422, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, and the like.

[0051]

[0055] In some examples, one or more of the processors of processing system 400 may be based on the ARM or RISC-V instruction set.

[0052]

[0056] Processing system 400 also includes memory 424, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, and the like. In this example, memory 424 includes computer-executable components that may be executed by one or more of the aforementioned processors of processing system 400.

[0053]

[0057] In particular, in this example, memory 424 includes a temperature measurement component 424A, a temperature threshold exceedance determination component 424B, a processing unit selection component 424C, a scheduling component 424D, and a neural network 424E. The illustrated components, and other components not illustrated, may be configured to perform various aspects of the methods described herein.

[0054]

[0058] Generally, the processing system 400 and / or its components may be configured to perform the methods described herein.

[0055]

[0059] Notably, in other embodiments, aspects of the processing system 400 may be omitted, such as when the processing system 400 is a server computer, etc. For example, the multimedia processing unit 410, the wireless connectivity component 412, the sensor processing unit 416, the ISP 418, and / or the navigation processor 420 may be omitted in other embodiments. Additionally, aspects of the processing system 400, such as training models and using the models to generate inferences, such as user authentication predictions, may be distributed.

[0056] Example clauses

[0060] The following numbered clauses describe example implementations.

[0057]

[0061] Clause 1: A method implemented on a computing device having a plurality of processing units, the method comprising: measuring a temperature for each of a plurality of locations on the computing device during execution of operations in a first portion of a neural network on a first processing unit of the computing device; determining that the measured temperature for the first processing unit exceeds a threshold temperature; selecting a second processing unit of the computing device to use in executing operations in a second portion of the neural network based on one or more operating parameters for the computing device; and scheduling execution of the operations in the second portion of the neural network on the second processing unit.

[0058]

[0062] Clause 2: The method of clause 1, wherein the first portion of the neural network and the second portion of the neural network include layers of a neural network configured for execution on the same set of processing units.

[0059]

[0063] Clause 3: The method of clause 1 or 2, wherein a first portion of the neural network is a member of a first set of layers configured for execution on a first set of processing units of a computing device, and a second portion of the neural network is a member of a second set of layers configured for execution on a second set of processing units of the computing device.

[0060]

[0064] Clause 4: The method of clause 3, wherein the first set of layers includes a set of layers configured with a first set of quantization parameters.

[0061]

[0065] Clause 5: The method of clause 4, wherein the second set of layers includes a set of layers configured with a second set of quantization parameters, the second set of quantization parameters corresponding to quantization over a smaller data type than the first set of quantization parameters.

[0062]

[0066] Clause 6: The method of any one of clauses 3 to 5, wherein the first set of processing units includes a neural processing unit (NPU), a digital signal processor (DSP), and multiple central processing unit (CPU) cores.

[0063]

[0067] Clause 7: The method of clause 6, wherein the second set of processing units includes multiple CPU cores and multiple graphics processing unit (GPU) processors.

[0064]

[0068] Clause 8: The method of any one of clauses 1 to 7, wherein selecting the second processing unit is further based on a ranking of types of processing units for performing operations within the second part of the neural network.

[0065]

[0069] Clause 9: The method of clause 8, wherein the ranking of the types of processing units is based on the size of data to be processed using the second part of the neural network and the performance level associated with each type of processing unit within the computing device.

[0066]

[0070] Clause 10: The method of any one of clauses 1 to 9, wherein the one or more operating parameters include one or more of a distance between the one or more processing units and the first processing unit, a temperature of the one or more processing units, or a current load on the one or more processing units.

[0067]

[0071] Clause 11: The method of clause 10, wherein selecting the second processing unit includes selecting a processing unit furthest from the first processing unit having a measured temperature below the threshold temperature.

[0068]

[0072] Clause 12: The method of clause 10 or 11, wherein selecting a second processing unit includes identifying a set of processing units having a distance from the first processing unit that exceeds a distance threshold and a measured temperature that is below a threshold temperature, and selecting a second processing unit from the identified set of processing units.

[0069]

[0073] Clause 13: The method of clause 12, wherein identifying the set of processing units further comprises identifying processing units having a current load below a threshold load.

[0070]

[0074] Clause 14: A processing system comprising a memory containing computer-executable instructions and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any one of clauses 1 to 13.

[0071]

[0075] Clause 15: A processing system comprising means for carrying out the method according to any one of clauses 1 to 13.

[0072]

[0076] Clause 16: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any one of clauses 1 to 13.

[0073]

[0077] Clause 17: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any one of clauses 1 to 13.

[0074] Additional Considerations

[0078] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples described herein are not intended to limit the scope, applicability, or embodiments described in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of the elements described without departing from the scope of the disclosure. Various embodiments may omit, substitute, or add various procedures or components as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented and a method may be performed using any number of aspects described herein. Moreover, the scope of the disclosure is intended to encompass apparatuses or methods that are implemented using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0075]

[0079] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0076]

[0080] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0077]

[0081] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0078]

[0082] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of those methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order of specific steps and / or actions and / or the use of those steps and / or actions may be modified without departing from the scope of the claims. Furthermore, various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. Those means may include various hardware and / or software component(s) including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors, and / or various hardware and / or software module(s). In general, when operations are illustrated in figures, those operations may have corresponding equivalent means-plus-function components that are similarly numbered.

[0079]

[0083] The following claims are not intended to be limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. Within the claims, reference to an element by the singular is not intended to mean "only one" unless expressly stated as such, but rather "one or more." Unless otherwise expressly stated, the term "several" refers to one or more. No element of a claim is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means of" or, in the case of a method claim, unless the element is recited using the phrase "step of." All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. 1. A method executed on a computing device having a plurality of processing units, comprising: Measuring a temperature for each of a plurality of locations on the computing device during execution of operations within a first portion of a machine learning model on a first processing unit of the computing device; determining that a measured temperature for the first processing unit exceeds a threshold temperature; Selecting a second processing unit of the computing device to use in performing operations in a second portion of the machine learning model based on one or more operating parameters for the computing device; scheduling execution of operations within the second portion of the machine learning model on the second processing unit; A method comprising:

2. 2. The method of claim 1 , wherein the first portion of the machine learning model and the second portion of the machine learning model comprise layers of a neural network configured for execution on a same set of processing units.

3. 2. The method of claim 1 , wherein the first portion of the machine learning model is a member of a first set of layers configured for execution on a first set of processing units of the computing device, and the second portion of the machine learning model is a member of a second set of layers configured for execution on a second set of processing units of the computing device.

4. The method of claim 3 , wherein the first set of layers comprises a set of layers configured with a first set of quantization parameters.

5. the second set of layers includes a set of layers configured with a second set of quantization parameters; The method of claim 4 , wherein the second set of quantization parameters corresponds to quantization across a smaller range of data types than the first set of quantization parameters.

6. 4. The method of claim 3, wherein the first set of processing units includes a neural processing unit (NPU), a digital signal processor (DSP), and multiple central processing unit (CPU) cores.

7. The method of claim 6 , wherein the second set of processing units includes the multiple CPU cores and multiple graphics processing unit (GPU) processors.

8. 2. The method of claim 1, wherein selecting the second processing unit is further based on a ranking of types of processing units for performing operations in the second portion of the machine learning model.

9. 9. The method of claim 8, wherein the ranking of types of processing units is based on a size of data to be processed with the second portion of the machine learning model and a performance level associated with each type of processing unit in the computing device.

10. 2. The method of claim 1 , wherein the one or more operating parameters include one or more of a distance between one or more processing units and the first processing unit, a temperature of the one or more processing units, or a current load on the one or more processing units.

11. The method of claim 10 , wherein selecting the second processing unit comprises selecting a processing unit furthest from the first processing unit having a measured temperature below a threshold temperature.

12. Selecting the second processing unit identifying a set of processing units having a distance from the first processing unit that exceeds a distance threshold and a measured temperature that is below a threshold temperature; selecting the second processing unit from the identified set of processing units; The method of claim 10, comprising:

13. The method of claim 12 , wherein identifying the set of processing units further comprises identifying processing units having a current load below a threshold load.

14. 1. A processing system comprising: a memory having computer-executable instructions; One or more processors for executing the computer-executable instructions and for providing the processing system with: measuring a temperature for each of a plurality of locations on a computing device during execution of operations within a first portion of the machine learning model on a first processing unit of the computing device; determining that the measured temperature of the first processing unit exceeds a threshold temperature; selecting a second processing unit of the computing device to use in performing operations in a second portion of the machine learning model based on one or more operating parameters for the computing device; one or more processors configured to schedule execution of operations within the second portion of the machine learning model on the second processing unit; A processing system comprising:

15. 15. The processing system of claim 14, wherein the first portion of the machine learning model and the second portion of the machine learning model comprise layers of a neural network configured for execution on a same set of processing units.

16. 15. The processing system of claim 14, wherein the first portion of the machine learning model is a member of a first set of tiers configured for execution on a first set of processing units of the computing device, and the second portion of the machine learning model is a member of a second set of tiers configured for execution on a second set of processing units of the computing device.

17. 20. The processing system of claim 16, wherein the first set of layers comprises a set of layers configured with a first set of quantization parameters.

18. the second set of layers includes a set of layers configured with a second set of quantization parameters; 20. The processing system of claim 17, wherein the second set of quantization parameters corresponds to quantization across a smaller range of data types than the first set of quantization parameters.

19. 20. The processing system of claim 16, wherein the first set of processing units includes a neural processing unit (NPU), a digital signal processor (DSP), and multiple central processing unit (CPU) cores.

20. 20. The processing system of claim 19, wherein the second set of processing units comprises the multiple CPU cores and multiple graphics processing unit (GPU) processors.

21. 15. The processing system of claim 14, wherein the processor is configured to select the second processing unit further based on a hierarchy of processing unit types for performing operations in the second portion of the machine learning model.

22. 22. The processing system of claim 21 , wherein the ranking of types of processing units is based on a size of data to be processed with the second portion of the machine learning model and a performance level associated with each type of processing unit in the computing device.

23. 15. The processing system of claim 14, wherein the one or more operating parameters include one or more of a distance between one or more processing units and the first processing unit, a temperature of the one or more processing units, or a current load on the one or more processing units.

24. 24. The processing system of claim 23, wherein to select the second processing unit, the processor is configured to select a processing unit that is furthest away from the first processing unit having a measured temperature below a threshold temperature.

25. To select the second processing unit, the processor: identifying a set of processing units having a distance from the first processing unit that exceeds a distance threshold and a measured temperature that is below a threshold temperature; 24. The processing system of claim 23, configured to select the second processing unit from the identified set of processing units.

26. 26. The processing system of claim 25, wherein to identify the set of processing units, the processing system is further configured to identify processing units having a current load below a threshold load.

27. 1. A processing system comprising: means for measuring a temperature for each of a plurality of locations on a computing device during execution of operations within a first portion of the machine learning model on a first processing unit of the computing device; means for determining that the measured temperature of the first processing unit exceeds a threshold temperature; means for selecting a second processing unit of the computing device to use in performing operations in a second portion of the machine learning model based on one or more operating parameters for the computing device; means for scheduling execution of operations within the second portion of the machine learning model on the second processing unit; A processing system comprising:

28. When executed by one or more processors of a processing system, the processing system comprises: Measuring a temperature for each of a plurality of locations on a computing device during execution of operations within a first portion of the machine learning model on a first processing unit of the computing device; determining that a measured temperature for the first processing unit exceeds a threshold temperature; Selecting a second processing unit of the computing device to use in performing operations in a second portion of the machine learning model based on one or more operating parameters for the computing device; scheduling execution of operations within the second portion of the machine learning model on the second processing unit; A non-transitory computer-readable medium comprising computer-executable instructions for performing a method, comprising:

Images