Predicting frequency and power of integrated hardware circuits using machine learning.

Abstract

A method and system (including a computer-readable medium) for performing frequency and power predictions for an integrated circuit in a system-on-a-chip ("SoC") is described. The system detects that an application is running on the SoC and determines the application's workload type based on indicators generated concurrently with the application's execution. Inferences are computed by the SoC's DVFS prediction engine based on the workload type and indicators. These inferences are used to achieve threshold quality of service ("QoS") when the application runs on the SoC. The DVFS prediction engine generates predicted frequency and power values ​​from the computed inferences. These predicted frequency or power values ​​are used to adjust the operating frequency or power output in the hardware integrated circuit to achieve threshold QoS when the application runs on the SoC.

Description

Background Art

[0001] This specification generally relates to machine learning and integrated circuits.

[0002] A machine learning model can employ a neural network having one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks can be a convolutional neural network (CNN) configured for image processing, or a recurrent neural network (RNN) configured for audio and language processing.

[0003] Different types of machine learning architectures can be used to perform various tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering. A neural network layer can have a corresponding set of parameters or weights. Weights are used to process an input (e.g., a batch of inputs) through the neural network layer to generate a corresponding output of the layer for computing neural network inference. A batch of inputs and a set of kernels can be represented as respective tensors, i.e., as a first multi-dimensional array of inputs and a second, different multi-dimensional array of weights.

[0004] A hardware accelerator is an application-specific integrated circuit for executing a neural network or other machine learning model. The integrated circuit can include memory used to store data for a plurality of tensors. The memory includes individual memory locations identified by unique addresses (e.g., virtual addresses or physical addresses). The address locations can correspond to elements of a tensor. Data corresponding to elements of one or more tensors can be traversed or accessed using the control logic of the integrated circuit.

Summary of the Invention

[0005] This specification describes machine learning (ML) techniques and corresponding predictive models configured to generate frequency and / or power predictions for operating individual components and processor devices of a system-on-a-chip ("SoC"). The predictive ML model(s) generates frequency or power predictions based on the detected workload type of the application and relevant indicators processed when the application runs using the SoC. The frequency and power predictions establish dynamic voltage-frequency scaling ("DVFS") values ​​that are applied to the components of the SoC to improve or enhance the overall performance of the SoC.

[0006] The SoC's DVFS prediction engine includes, or communicates with, a workload detector and a predictive ML model for running a comprehensive DVFS framework that leverages adaptive learning to adjust DVFS settings on the SoC. The workload detector generates the workload type of the application running on the user device. The workload type (e.g., gaming) is passed to the predictive ML model of the DVFS framework to compute inferences used to achieve threshold quality of service ("QoS") when the application runs on the SoC. From the computed inferences, the DVFS prediction engine generates predicted frequency and power values.

[0007] The DVFS prediction engine uses its predictions to more efficiently adjust DVFS settings, for example, to achieve the operating point with the fastest frequency and lowest power consumption. For example, advancing scene changes in a first-person gaming application running on a user device may require running graphics-intensive workloads on the SoC. The DVFS prediction engine can be used to determine the operating point on the SoC that maximizes frequency and duty cycle, while simultaneously optimizing QoS and minimizing power consumption. By using DVFS settings, it is possible to achieve performance threshold levels while minimizing the resource overhead required to achieve those performance levels.

[0008] One aspect of the subject matter described herein can be embodied in a computer-based method for predicting frequency and power in a hardware integrated circuit. The method includes detecting that an application is running on a System of Control (SoC) of a hardware integrated circuit and determining the workload type of the application based on indicators generated concurrently with the application's execution. Based on the workload type and indicators, the method includes computing ML inference using the DVFS prediction engine of the SoC. The ML inference is used to achieve threshold QoS when the application is running on the SoC. From the computed inference, the DVFS prediction engine generates predicted frequency and predicted power values. The method further includes using the predicted frequency or power values ​​to adjust the operating frequency or power output of the hardware integrated circuit to achieve threshold QoS when the application is running on the SoC.

[0009] These and other embodiments may optionally include one or more of the following features. For example, in some embodiments, generating predicted frequency and power values ​​from computed inference includes the DVFS prediction engine generating minimum frequency and corresponding minimum power values ​​for achieving threshold QoS when the application runs on the SoC. In some embodiments, generating predicted frequency and power values ​​from computed inference includes the DVFS prediction engine generating maximum frequency and corresponding minimum power values ​​required to run at the maximum operating frequency in the hardware integrated circuit, based on the maximum frequency value.

[0010] Computing inference may include computing inference using the frequency prediction model of the DVFS prediction engine based on the workload type and indicators, and computing inference using the power prediction model of the DVFS prediction engine based on the workload type and indicators. In some embodiments, the SoC is integrated within the user device, and the frequency prediction model is trained and executed on the SoC. The power prediction model may be a pre-trained neural network model, and the power prediction model may be trained offline separately from the SoC and user device and then loaded onto the SoC for execution on the user device.

[0011] In some embodiments, the SoC is for a user device, and the method further includes: i) determining a first workload type for a first application to run on the user device; ii) adjusting the operating frequency and power output of a hardware integrated circuit using predicted frequency and power values ​​generated from inferences calculated by a DVFS prediction engine based on the first workload type; iii) determining a second workload type for a second application to run on the user device; and iv) adjusting the operating frequency and power output of a hardware integrated circuit using predicted frequency and power values ​​generated from inferences calculated by a DVFS prediction engine based on the second workload type.

[0012] The method further includes adjusting the operating frequency or power output with reference to a first workload type to achieve a first threshold QoS when a first application runs on the SoC, and adjusting the operating frequency or power output with reference to a second workload type to achieve a second threshold QoS when a second application runs on the SoC. In some embodiments, the first and second workload types are different workload types, and the first and second applications are the same application. In some embodiments, computing inference includes computing inference based on the workload type, a number of software indicators, and a number of hardware indicators.

[0013] This aspect and other embodiments of other aspects include corresponding systems, devices, and computer programs encoded on a computer storage device and configured to perform actions of the method. One or more computer systems can be configured in this way by software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform actions when in operation. One or more computer programs can be configured in this way by having instructions that cause the device to perform actions when executed by a data processing device.

[0014] The subject matter described herein may be implemented in particular embodiments to achieve one or more of the following advantages:

[0015] The disclosed technology can provide fine-grained control of frequency and power output to efficiently adjust DVFS settings on the SoC of a user device such as a tablet or smartphone. The frequency and / or power prediction models of the DVFS prediction engine can predict, estimate, or otherwise determine the respective frequency and power output values ​​when an application is launched on the user device. In connection therewith, the DVFS prediction engine can estimate, predict, or otherwise determine QoS requirements such as frame rate (FPS), latency, and system-level power limits based on inferences calculated by the frequency prediction model, the power prediction model, or both.

[0016] The DVFS prediction engine can run a comprehensive DVFS framework that leverages adaptive learning to adjust DVFS settings on the SoC. Predicted frequency and power values ​​are passed to the DVFS framework and used to more efficiently adjust DVFS settings to achieve an operating point that enables high-speed and power-efficient performance, for example. For example, when running an application, the frequency and power prediction model is used to determine the operating point on the SoC that optimizes quality of service while minimizing power consumption. Thus, DVFS settings are used to maximize the performance of the integrated circuit on the SoC while also minimizing the resource overhead required to achieve that maximum performance.

[0017] Details of one or more embodiments of the subject matter described herein are given in the accompanying drawings and the following description. Other potential features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. [Brief explanation of the drawing]

[0018] [Figure 1] This is a block diagram of an exemplary computing system with a system-on-a-chip architecture. [Figure 2] An example of a workload type detection algorithm is shown. [Figure 3] This diagram illustrates an example of determining service quality deviations for a specific workload type. [Figure 4] This exhibits an exemplary adaptive ML training / self-learning module. [Figure 5] Figure 1 shows an illustrative process for workload discovery using the computing system. [Figure 6] An exemplary frequency prediction model is shown. [Figure 7] An exemplary power prediction model is shown. [Figure 8] This is an illustrative process flow diagram for training frequency and power prediction models. [Figure 9]An exemplary process for predicting frequency and power in a hardware integrated circuit.

Best Mode for Carrying Out the Invention

[0019] Like reference numerals and designations in the various drawings indicate like elements.

[0020] FIG. 1 is a block diagram of an exemplary computing system 100 that includes a system-on-chip 102 (“SoC102”). SoC102 includes a central processing unit 104 (“CPU104”), a dynamic voltage and frequency scaling (“DVFS”) prediction engine 106, and a circuit block 108. CPU104 generates one or more indicators 105, such as an app launch indicator, or a function call triggered in response to the execution or launch of an application on a user device. CPU104 also generates one or more application values 107. Application values 107 can be associated with a function call, can be a description of an event that occurs during the execution of an application, or both.

[0021] CPU104 can be a general-purpose CPU (e.g., a single-core or multi-core CPU). In some embodiments, DVFS prediction engine 106 is implemented as a software module of CPU104 that uses one or more hardware resources of CPU104. In some examples, CPU104 is configured as an instruction and vector data processing engine that processes data obtained from the system memory of SoC102. At least one processor of circuit block 108 can also include, or be configured to include, instructions and vector data engines that operate on vectors and matrices of data and / or operands.

[0022] The DVFS prediction engine 106 is configured to include or access a workload detector 110, an adaptive ML training / self-learning module 112 (the "adaptive learning module 112"), and a DVFS prediction model 113. The DVFS prediction engine 106 can be implemented in software, hardware, or both. Aspects of the DVFS prediction engine 106 can also be implemented as firmware of the SoC 102 or as devices of the SoC 102 such as the CPU 104 or circuit blocks 108. As will be described in detail below, the DVFS prediction engine 106 is configured to generate control signaling for dynamically adjusting voltage (or current) and frequency settings (the "DFVS settings") throughout the component system 100.

[0023] The workload detector 110 receives inputs 114 corresponding to the indicator 105 and the application value 107. The workload detector 110 processes the inputs 114 using an exemplary workload detection algorithm 116 to generate one or more workload types 118. This will be described below with reference to FIG. 2. The adaptive learning module 112 iteratively adjusts the QoS parameters of the DVFS prediction engine 106 using an adaptive cost function. This will be described in more detail below with reference to FIGS. 3 and 4.

[0024] The DVFS prediction model 113 includes a predictive ML model that communicates with the CPU 104 and the workload detector 110 and utilizes an adaptive learning algorithm 115, and executes a comprehensive DVFS framework for adjusting DVFS settings in the SoC 102. For example, the DVFS prediction model 113 includes a frequency prediction ML model that predicts and / or generates frequency values, and a power prediction ML model that predicts and / or generates power values / settings. In some embodiments, the predicted frequency and power values define one or more DVFS settings in the SoC 102. The frequency prediction ML model and the power prediction ML model of the DVFS prediction model 113 will be described in further detail below with reference to FIGS. 6-8.

[0025] The workload detector 110 generates a workload type 118 for an application running on the user device. The workload type (e.g., gaming) and associated input 114 are passed to the DVFS framework's predictive ML model as input and corresponding output for computing inference. The calculations and corresponding outputs are used to achieve threshold quality of service ("QoS") when the application runs on the SoC 102. For example, the DVFS predictive model 113 generates predicted frequency values ​​and predicted power values ​​as outputs from the computed inference. The DVFS predictive model 113 uses its predictions to more efficiently adjust the DVFS settings to achieve, for example, the lowest and fastest operating point for various IP devices on the SoC 102.

[0026] Circuit block 108 may be an intellectual property (IP) circuit block comprising one or more IP devices. Therefore, circuit block 108 may alternatively be referred to as IP block 108, and an IP block(s) may include one or more unique IP hardware devices / components. For example, IP / circuit block 108 may include a graphics processing unit (GPU) 120, a tensor processing unit (TPU) 122, a digital signal processor (DSP) 124, and an image signal processor (126). In some embodiments, a memory device 128 is also included among the devices of IP circuit block 108. In some embodiments, each of the GPU 120, TPU 122, DSP 124, ISP 126, and memory device 128 may be a unique IP block of a particular entity or device manufacturer.

[0027] In the example in Figure 1, the memory device 128 is shown as separate from the SoC 102 and the IP block 108. However, the configuration of the memory device 128 may be i) local to the IP block 108 and the SoC 102, ii) external to the IP block 108 and the SoC 102, or iii) both. In some embodiments, the memory device 128 is an exemplary random access memory of the SoC 102, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM. In some other embodiments, the memory device 128 may be, or include, various other types of memory, such as high-bandwidth memory (HBM), narrow-bandwidth memory (for storing, e.g., 8-bit values), or wide memory (for storing, e.g., 16-bit or 32-bit values).

[0028] In the example in Figure 1, System 100 and SoC 102 are integrated circuits for exemplary user / client devices, consumer electronic devices, or mobile devices, each of which may include items such as a smartphone 130a, a tablet 130b, a laptop 130c, a smartwatch, or a wearable device 130d. The devices may also include other items such as an e-notebook, a netbook, a smart speaker, or a mobile computer. In some embodiments, System 100, including SoC 102, is an integrated circuit for a desktop computer, a network server, or an associated cloud-based asset.

[0029] The DVFS prediction engine 106 dynamically controls DVFS settings across the SoC 102, including the CPU 104 and IP block 108 devices, using operations performed by the workload detector 110, the adaptive learning module 112, and the DVFS prediction model 113. More specifically, the DVFS prediction engine 106 is configured to generate control signaling 119 and to use one or more discrete control values ​​of the control signaling 119 to adjust at least voltage, current, and frequency values ​​for the DVFS settings of at least the IP block 108, the CPU 104, the memory device 128, or a combination thereof.

[0030] For example, the control signaling 119 may include i) at least one control value for adjusting the DVFS setting in the CPU 104, ii) at least one control value for adjusting the DVFS setting in the GPU 120, iii) at least one control value for adjusting the DVFS setting in the TPU 122, iv) at least one control value for adjusting the DVFS setting in the DSP 124, and v) at least one control value for adjusting the DVFS setting in the ISP 126. In connection with this, the control signaling 123 may include at least one control value for adjusting the DVFS setting in the memory device 128. In some embodiments, each processor of the SoC 102 (e.g., CPU, GPU, or TPU) includes multiple cores, and the DVFS prediction engine 106 can generate control signaling 119 to adjust or adjust the DVFS setting in each core of the processor.

[0031] Figure 2 shows an exemplary workload type detection algorithm 116 ("Algorithm 116") performed by the workload detector 110. The workload detector 110 can detect or determine the workload type based on a static algorithm, a machine learning ("ML") algorithm, or both. Whether static or ML-based, the workload detector 110 leverages a unique detection algorithm 116 to generate an output specifying the detected workload type 118.

[0032] In some embodiments, algorithm 116 represents a process, procedure, or set of rules for performing calculations or other problem-solving operations performed by the DVFS prediction engine 106, the workload detector 110, or both. As shown below, the process, procedure, or set of rules may correspond to one or more computational flows. Insofar as algorithm 116 is described as something that takes action, for example, detecting, determining, executing, or generating, such actions are performed based on the execution of algorithm 116 by the DVFS prediction engine 106, the workload detector 110, or other devices of system 100.

[0033] As described above, the workload detector 110 processes the input 114 according to algorithm 116 to generate one or more workload types 118. Generally, the workload detector 110 initiates its operation to detect workload types in response to detecting or determining that an application is launched or running on the user device 130. Detectable workload types include compute workload types, high-interaction gaming workload types, low-interaction gaming workload types, user interface workload types, and background workload types. Other workload types, such as memory-intensive workloads, streaming workloads, or generative AI workloads, are also within the scope of this disclosure.

[0034] The type of workload detected by the workload detector 110 may be a workload that is processed by the system 100 to produce application output, such as graphic content, which is rendered on the display of the user device 130. For example, the application may be a gaming application, such as a first-person shooter game or a first-person racing / driver game. In this example, the gaming sequence of the application is rendered on the display of the user device 130 using one or more resources of the IP block 108.

[0035] Algorithm 116 may include one or more compute flows 202, 204, 206 for at least one subset of workload types detectable by the workload detector 110. In compute flow 202, algorithm 116 determines whether a first API function call is detected or triggered (208). The first API function call may be one or more software indicators (or software hints) indicating that a particular workload is computationally intensive, sensitive to high / long latency, and / or needs to be executed at low latency or within a threshold duration. The first API function call may include one or more of the OpenCL API calls, OpenGL API calls, or Vulkan API calls.

[0036] Generally, OpenCL is an API call that can be issued by SoC102 to perform calculations using one or more computing resources of System 100, such as TPU122, while OpenGL and Vulkan are their respective API calls that can be issued by SoC102 to perform specific graphics operations using one or more graphics processing resources of System 100, such as GPU120. For example, OpenGL is applied by SoC102 to generate user interface (UI) animations, manage embedded video functionality, or construct vector graphics for rendering visual elements of a given application.

[0037] API function calls may be managed by CPU 104. In some embodiments, each API function call is either a software indicator generated by CPU 104, detected by CPU 104, or both, or corresponds to one. Thus, algorithm 116 can make a decision regarding a first API function call based on one or more indicators 105 detected by CPU 104. If algorithm 116 determines that a first API function call was detected or triggered, algorithm 116 determines that the workload is a computationally intensive workload (210). If algorithm 116 determines that a first API function call was not triggered, algorithm 116 proceeds to computation flow 204.

[0038] In response to determining that the workload is a computationally intensive workload, algorithm 116 causes the DVFS prediction engine 106 or CPU 104 to generate a control signal 212. More specifically, algorithm 116 (or workload detector 110) may determine that the application workload is a computational workload 216, based on the determination that a first API function call has been detected, in response to the generation of the control signal 212, or both.

[0039] The control signal 212 is used to boost the operating frequency of at least one or more of the CPU 104, GPU 120, or both (214). In some embodiments, the control signal 212 is used to boost the operating frequency of each of the other resources of the SoC 102. For example, the control signal 212 can be passed to the CPU 104 and / or GPU 120 according to the frequency and power predictions generated by the DVFS prediction model 113 to boost the operating frequency of each of those devices. In some embodiments, the control signal 212 is passed to individual frequency and power prediction models to generate their respective minimum frequency and power setpoints to perform the corresponding operating frequency boost.

[0040] In some embodiments, using algorithm 116, the DVFS prediction engine 106 determines whether a computationally intensive workload is latency-critical or involves one or more operations that are latency-critical. In this embodiment, control signal 212 is passed to a resource of the SoC 102 (e.g., GPU 120) corresponding to a DVFS control signal that boosts a specific DVFS setting (e.g., frequency) of that resource. The DVFS prediction engine 106 boosts the specific DVFS setting to ensure that a latency-critical computationally intensive workload can meet or exceed the latency required when the workload is executed by the SoC 102 (e.g., low latency).

[0041] In computation flow 204, algorithm 116 determines whether a second API function call has been detected or triggered (220). The second API function call may be one or more software indicators (or software hints) that indicate a particular workload is graphics-intensive, are highly sensitive to lost frames or frame rates, and / or require rendering graphic content with sufficient detail, resolution, or both. The second API function call may include a textureBind API call or other related API call used to render graphic content, including textures and other details of its graphic content. In some embodiments, the second API function call(s) may include one or more of the aforementioned API function calls by reference to the first API function call.

[0042] Generally, textureBind is an API call that can also be issued on SoC102 to perform specific graphics operations using one or more graphics processing resources of System 100, such as GPU120. More specifically, textureBind can refer to or indicate texture changes in graphics content generated for an application, or scenes or avatars that trigger texture changes. Texture changes are signaled by textureBind calls, and one or more values ​​of textureBind calls are used to determine and / or generate high interaction parameter signals indicating high interaction gaming modes.

[0043] In some embodiments, a high interaction parameter signal is used or processed by the workload detector 110 to indicate that one or more gaming scenes or gaming content have a high FPS requirement, or at least one FPS that must exceed a certain threshold. Examples of high FPS requirements can vary depending on the type of user device 130 (e.g., smartphone or laptop), as well as other factors such as the screen refresh rate of the user device 130 or the resolution of the device's display. The maximum FPS of the user device 130 may be limited by the screen refresh rate of the device's display screen. For example, if the user device 130 is a mobile phone / smartphone, the typical refresh rate of its screen may be in the range of 240Hz or 120Hz. Thus, in this example, the maximum FPS would be 120 or 240. In some cases, a high FPS is determined for a given resolution, such as 1280x800 or 1280x32.

[0044] The workload detector 110 uses algorithm 116 to perform a comparison operation that evaluates an exemplary incoming textureBind value against a threshold. In some embodiments, the threshold is determined adaptively or dynamically by system 100. The exemplary textureBind threshold may vary depending on the gaming application or workload (e.g., 1000 or 1200). The comparison operation is performed to determine whether the application, or the workload associated with the application, is a high-interaction gaming workload or a low-interaction gaming workload. A second API function call may also be managed by CPU 104. In some embodiments, the second API function call is a software indicator that is generated by CPU 104, detected by CPU 104, or both, or corresponds to one. Thus, algorithm 116 can make a decision regarding a second, different API function call based on one or more indicators 105 detected by CPU 104.

[0045] If algorithm 116 determines that a second API function call was detected or triggered, algorithm 116 determines that the workload is a gaming workload (222). If algorithm 116 determines that no textureBind API call was triggered, algorithm 116 proceeds to computation flow 206.

[0046] In response to determining that the workload is a gaming workload, algorithm 116 determines whether the textureBind value exceeds a threshold. This determination is described above with reference to the comparison operation with the incoming textureBind value and the textureBind threshold. If algorithm 116 determines that the textureBind value exceeds the threshold, algorithm 116 detects or determines that the workload is a high-interaction gaming workload 226. However, if algorithm 116 determines that the textureBind value does not exceed the threshold, algorithm 116 detects or determines that the workload is a low-interaction gaming workload 228.

[0047] When the workload detector 110 determines that the workload is a high-interaction gaming workload 226, the algorithm 116 causes the DVFS prediction engine 106 or CPU 104 to generate a control signal that boosts at least one or more operating frequencies of the GPU 120. The control signal may also be used to boost the operating frequencies of other resources of the SoC 102. In some cases, this control signal corresponds to the high-interaction parameter signal described above.

[0048] In some embodiments, using algorithm 116, the DVFS prediction engine 106 determines whether the high-interaction gaming workload 226 is a frame-critical workload, a frame-rate critical workload, or involves one or more actions that are particularly sensitive to missing, dropped, or lost frames. In this embodiment, the high-interaction parameter signal is passed to a resource on the SoC 102 (e.g., GPU 120) to correspond to a DVFS control signal that boosts a specific DVFS setting (e.g., frequency) for that resource. The DVFS prediction engine 106 boosts the specific DVFS setting to ensure that the frame-critical, high-interaction workload can meet or exceed the target requirements for lost frames when the workload is executed by the SoC 102.

[0049] In the calculation flow 206, algorithm 116 determines whether the display_on signal has been detected or triggered (230). In some embodiments, the display_on signal is a software indicator that is generated by the CPU 104, detected by the CPU 104, or both, or corresponds to such an indicator. Thus, algorithm 116 can make a decision regarding the display_on signal based on one or more indicators 105 detected by the CPU 104. For example, the display_on signal may be a software indicator (or software hint) that indicates the device state of the user device 130, such as whether the screen or display of the user device is on or off.

[0050] If the workload detector 110 determines that a display_on signal has been detected or triggered, algorithm 116 determines whether the detected memory allocation is greater than a threshold memory allocation (232). If algorithm 116 determines that a display_on signal was not detected or triggered, algorithm 116 causes the workload detector 110 to generate an output indicating that the detected workload type is a background workload. This is described in more detail below.

[0051] If the detected memory allocation exceeds a threshold memory allocation, the workload detector 110 determines that the workload associated with the application is a memory-intensive (234) user interface (UI) workload. If the detected memory allocation does not exceed a threshold memory allocation, the workload detector 110 (or algorithm 116) determines that the workload associated with the application is a non-memory-intensive UI workload (236), such as a workload that generates UI elements that are not particularly memory-intensive.

[0052] In some embodiments, to determine whether a workload is memory-intensive, the workload detector 110 receives a memory allocation input (e.g., mem_alloc), analyzes or uses the memory allocation input to determine whether the workload is memory-intensive or non-memory-intensive. The memory allocation input may be found in a hardware / software indicator 105 detected (or generated) by the CPU 104. In some embodiments, the memory allocation information is provided by or obtained from the operating system (OS) kernel.

[0053] If the workload detector 110 determines that the workload or application is memory-intensive (234), the workload detector 110 (or algorithm 116) causes the DVFS prediction engine 106 or CPU 104 to generate a control signal that boosts the operating frequency of at least one or more memory devices 128 (238). This control signal can also be used to boost the operating frequency of other resources of the SoC 102. For example, this control signal can be used to boost the frequency of a memory interface, which can facilitate the retrieval of specific data, such as data generated by the CPU 104 for a particular gaming application driver.

[0054] In some embodiments, the control signal that boosts the operating frequency of the memory device 128 represents a RAM / DDR frequency boost signal, and the DVFS prediction engine 106 (or CPU 104) can generate multiple memory frequency boost signals. The memory device 128 may include multiple DDR / DRAM modules, and each RAM / DDR boost signal can be used to boost the operating frequency of the corresponding DDR / DRAM module. For example, the operating frequency of the memory device 128 can be boosted from 900MHz to 1.2GHz.

[0055] In terms of context, certain applications launched on user device 130, such as gaming apps, may have processing and graphics resource requirements that at times bottleneck CPU 102. For example, CPU 102 may manage the execution or operation of the gaming application's driver, and CPU 102 may be both a consumer and a generator of the data necessary to run the application (or workload) on user device 130. If CPU 104 is the data generator, it may be necessary to transfer the data for initial or further processing via the resources of IP block 108, such as GPU 120.

[0056] If a particular workload or application is determined to be memory-intensive, the CPU 104 may be required to generate data (e.g., large amounts of data) corresponding to multiple memory allocations and supply it to specific resources in the IP block 108. In some embodiments, for certain graphical user interface (GUI) operations that require fast CPU response times, the CPU 104 may need to generate large amounts of data and supply it quickly to the GPU 120 or TPU 122.

[0057] In this embodiment, if the CPU 104 is too slow to supply information to the GPU, the CPU 104 may become a bottleneck in the computation flow. This is especially true for memory-intensive workloads or applications. Therefore, as described above, using algorithm 116, the workload detector 110 causes the DVFS prediction engine 106 to generate a control signal that boosts or otherwise adjusts the operating frequency of one or more memory modules in the memory device 128 (238). The frequency can be increased to facilitate the acquisition and transfer of data necessary to run the memory-intensive workload of the application.

[0058] Generally, launching an application on the user device 130 triggers or requires the allocation of memory resources such as buffers, registers, caches, and main memory. For example, in response to the detection of the app_launch signal, the CPU 104 can use allocation signals (e.g., alloc pages) to process the application's request for memory. In some embodiments, the memory allocation required to run the application workload is proportional to the number of API function calls. The workload detector 110 can determine a weight between at least the relative size of the requested memory allocation and the number of API function calls, and can determine the workload type based on the determined weight.

[0059] As described above, if algorithm 116 determines that the display_on signal was not detected or triggered, algorithm 116 transitions from computation flow 206 to background computation flow 240, causing the workload detector 110 to generate output specifying that the detected workload type is background workload 242. In some embodiments, background workload 242 includes one or more background tasks. Background workload 242 may be an exemplary non-GUI workload that includes the display_off device state.

[0060] For example, while an application is running, its underlying background workload 242 may include tasks that can turn off the display or screen of the user device 130 without affecting the application, the workload, or the execution of individual tasks within the workload. In some embodiments, the application itself, or the tasks of the background workload 242, may be for receiving text messages, tracking the alarm status of an alarm clock, or some other actions that require power but minimal processing and memory resources.

[0061] Based on algorithm 116, the workload detector 110 can detect or determine that an application or background workload 242 is a power-critical application (or workload). For example, the workload detector 110 can detect or determine a device state indicator corresponding to the power state of the user device 130's display. In some embodiments, the bit value of the device state indicator indicates whether the application is a power-critical application. Using this detection, the DVFS prediction engine 106 can determine that establishing or maintaining a low-power state is a primary priority for the SoC 102.

[0062] The device state indicator is used to generate DVFS control values ​​by referencing background tasks of the application or workload. More specifically, the DVFS control signaling 119 may include DVFS control values ​​generated by referencing background tasks of the application based on the bit values ​​of the device state indicator. For example, the DVFS prediction engine 106 can generate DVFS control signals 119 that are passed to resources of the SoC 102 (e.g., IP block 108) to establish low-power DVFS settings (e.g., voltage, current, frequency, etc.) by, for example, reducing the operating voltage and frequency across the entire SoC 102.

[0063] Figure 3 shows an exemplary flowchart of a deviation module 300 used to determine QoS violations (or deviations) of a particular workload type. As described above, the workload detector 110 is configured to detect multiple types of workloads 118. For a given application, the workload detector 110 can detect or determine at least a compute workload 210, a high-interaction gaming workload 226, a low-interaction gaming workload 228, a UI / GUI workload 234, and a background workload 242.

[0064] The deviation module 300 includes comparator logic 302, which includes multiple comparators. More specifically, the comparator logic 302 includes a latency comparator 304, a first frame comparator 306, a second frame comparator 308, a UI response comparator 310, and a power comparator 312. The latency comparator 304 compares the detected or observed latency with the target latency and generates a QoS violation if the observed latency is less than the target. An exemplary target latency may be 200 milliseconds (ms) or 300 ms.

[0065] The first frame comparator 306 i) compares the detected or observed loss frame value to the target loss frame value, and ii) compares the detected or observed FPS to the target FPS (e.g., low). The first frame comparator 306 generates a QoS violation if the observed loss frame value is less than the target value and the observed FPS is not equal to the target FPS. Similarly, the second frame comparator 308 i) compares the detected or observed loss frame value to the target loss frame value, and ii) compares the detected or observed FPS to the target FPS (e.g., high). The second frame comparator 308 generates a QoS violation if the observed loss frame value is less than the target and the observed FPS is not equal to the target FPS.

[0066] The UI response comparator 310 compares the detected or observed UI response with the target response and generates a QoS violation if the observed UI response is less than the target. The power comparator 312 compares the detected or observed power value with the target power and generates a QoS violation if the observed power value is less than the target.

[0067] Figure 4 shows an exemplary adaptive ML training / self-learning module 112. The adaptive learning module 112 includes one or more machine learning models 402, a set of control values ​​404, and one or more adaptive QoS cost functions 406.

[0068] The QoS cost function(s) 406 is used to measure, evaluate, or otherwise determine the performance of an ML model on a given dataset. At least one of the machine learning models 402 can be implemented using an artificial neural network with multiple layers, including one or more hidden layers. In some embodiments, the QoS cost function(s) 406 quantifies the error between the predicted and expected values. The predicted and expected values ​​can be represented by the inference outputs of an ML model 402, such as a frequency prediction ML model and a power prediction ML model, as described below with reference to the examples in Figures 6 and 7, respectively.

[0069] For example, the DVFS prediction engine 106 is configured to compute an inference output using a workload detector 110, an adaptive learning module 112, or both. The inference output may include the detected workload type (e.g., high-interaction gaming workload) and several quality control values ​​404, such as a target value for frames per second (fps), a target frequency value, or a target power value. To compute the inference output, the DVFS prediction engine 106 can process the ML input corresponding to each function call indicator through a hidden layer (e.g., a neural network layer) of at least one ML model 402. For example, the ML input may be processed through the hidden layer according to an adaptive ML algorithm.

[0070] Quality control values ​​can be calculated by applying an adaptive ML algorithm to some (or all) of the hardware and software indicators 105, application values ​​107, or both. Quality control values ​​404 may be thresholds for implementing specific quality of service requirements, such as ensuring that image post-processing operations are computed as quickly as possible (e.g., with minimal latency), or that a gaming application iterates through multiple scene changes without dropping a single frame.

[0071] The adaptive learning module 112 can use an adaptive QoS cost function 406 to generate an output (e.g., a real number) representing a quantified error. The adaptive learning module 112 can also use the QoS cost function 406 to iteratively adjust control values ​​for the QoS parameters of the DVFS prediction engine 106. The control values ​​include target values ​​and corresponding thresholds for each target value. In one or more examples, the target value and the threshold may be the same value. The QoS parameters correspond to one or more detectable workload types and include parameters such as latency, power, lost / missing frames, frame rate, and UI response time. Application performance and other parameters related to determining DVFS settings are also within the scope of this disclosure.

[0072] The adaptive learning module 112 uses the QoS cost function 406 to calculate the deviation score for each QoS parameter. For example, each deviation score may be calculated based on a specific quality control value 404 corresponding to the QoS parameter, and at least one of i) one or more function call indicators, and ii) one or more application values ​​107. Using the self-learning loop of the adaptive learning module 112, an inference output is calculated that includes one or more previously updated control values ​​404 based on i) the adaptive cost function 406 and ii) at least one of the deviation scores for each specific QoS parameter.

[0073] The adaptive learning module 112 may include multiple QoS cost functions 406, each QoS cost function 406 may correspond to one or more ML models 402. For example, the adaptive learning module 112 may include an ML model 402 and an adaptive QoS cost function 406 for each workload type detectable by the workload detector 110. The adaptive learning module 112 is configured to dynamically determine control values, including thresholds, for each QoS parameter based on one or more ML models and the corresponding QoS cost functions 406. In some embodiments, the DVFS prediction engine 106 is configured as an integrated workload detection, frequency prediction, and power prediction ML model that generates multiple DVFS control values ​​based on one or more inference outputs.

[0074] Figure 5 shows an exemplary process 500 for workload discovery and DVFS configuration tuning using the computing system of Figure 1. One or more steps of process 500 are performed to determine one or more attributes of the application workload using the workload detector of the DVFS prediction engine. More specifically, workload discovery and DVFS configuration tuning are performed based on a workload discovery machine learning ("ML") model implemented in the workload detector 110, the DVFS prediction engine 106, or both.

[0075] Generally, process 500 can be implemented or executed using the system 100 and SoC 102 described above. Therefore, the description of process 500 can refer to the computing resources of system 100 and SoC 102 described above. In some examples, the steps or actions of process 500 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction can be stored in a non-temporary machine-readable memory device and can be executed by one or more of the processors or other resources described in this document, such as the CPU 104, the scalar cores or compute tiles of the TPU 122, a hardware ML accelerator, or a neural network processor.

[0076] In some embodiments, the steps of process 500 are performed by a hardware integrated circuit to generate an ML output, which includes the output of the neural network layers of a neural network running one or more machine learning (ML) models. For example, this output may be part of a computation for an ML task or inference workload to generate an image processing, speech processing, or image recognition output. As shown above, the integrated circuit may be a dedicated neural network processor or hardware ML accelerator configured to accelerate computations for generating different types of data processing outputs.

[0077] Referring again to process 500, system 100 receives a request to launch an application on a user device (502). More specifically, the SoC 102 of a user device such as user 130a / b / c receives a request to launch an application on user device 130. For example, user device 130 may receive an input request from the user to launch a gaming application such as Halo® or Call of Duty®. Based on the request, system 100 identifies, detects, or otherwise determines one or more software (and / or hardware) indicators 105 (504). Multiple software (and / or hardware) indicators 105 may form a set of inputs 114, including an app launch indicator and one or more function call indicators. As its name suggests, the app launch indicator indicates that an application has been launched on user device 130.

[0078] The workload detection ML model computes an inference output that includes the detected workload type and several quality control values ​​(506). For example, the workload detection ML model computes the inference output by applying the adaptive ML algorithm of the workload detection ML model to each of the indicators 105. In some embodiments, the workload detection ML model computes or generates the inference output using one or more of the computation flows 202, 204, and 206. For example, the output of computation flow 202 may be an indication that the application workload is computation workload 216. The workload detector 110 uses this indication to generate and output a workload_type signal that specifies the type of workload detected.

[0079] System 100 uses the inference output to generate a Dynamic Variable Frequency Signal ("DVFS") control value (508). In some embodiments, the workload detector 110 passes the detected workload type 118 as an output signal to the adaptive ML training / self-learning module 112. For example, the workload detector 110 can pass the detected workload type 118 along with one or more application values ​​107, or it can cause the CPU 104 to pass one or more application values ​​107. The DVFS prediction engine 106 uses the inference output to generate a DVFS control value.

[0080] The DVFS control value is for controlling the DVFS setting on the SoC 102 based on at least the detected workload type 118. For example, the workload detector 110 may determine that the workload associated with the application is a high-interaction gaming workload, and the system 100 may use the DVFS control value to control the DVFS setting of the GPU 120 so that frame loss on the user device 130 is minimized or eliminated as the application transitions through various high-interaction gaming scenarios. In some embodiments, the DVFS setting is controlled in a way that allows the user to play high-interaction gaming applications on the user device 130 with minimal (or zero) frame loss, while simultaneously adjusting the power output to maximize available battery life.

[0081] In another example, the workload detector 110 may determine that a workload associated with a music streaming application is a background workload capable of outputting an audio stream regardless of the device state associated with the user device 130's display / screen. The system 100 can minimize power consumption on the user device 130 by using DVFS control values ​​to control the DVFS settings of one or more processor cores of the SoC 102. In some embodiments, the DVFS settings are controlled in a way that allows the user to play high-interaction gaming applications on the user device 130 with minimal (or zero) frame loss.

[0082] Figure 6 shows an exemplary frequency prediction model 600. In some embodiments, the frequency prediction ML model 600 is included in the exemplary DVFS prediction model 113 described above with reference to Figure 1. As mentioned above, machine learning architectures are used to perform a variety of tasks related to classification or pattern recognition, prediction with data modeling, and information clustering. In the example in Figure 6, the frequency prediction ML model 600 employs one or more neural networks with multiple layers of nodes to generate an output, e.g., classification, for an incoming input. The neural network (or more) includes one or more hidden layers in addition to an input layer and a corresponding output layer.

[0083] In some embodiments, the memory neural network is a convolutional neural network (CNN) or a recurrent neural network (RNN). A neural network layer can have a corresponding set of parameters or weights. Weights are used to process inputs (e.g., batches of inputs) through the neural network layers (e.g., hidden layers) to generate the corresponding outputs of the layers for computing neural network inference. In some embodiments, the weights are updated during training using backpropagation techniques. The batches of inputs and kernels processed through the layers can be represented as tensors, such as a first multidimensional array of inputs and a second distinct multidimensional array of weights.

[0084] The frequency prediction ML model 600 is configured to generate an inference output from a set of received inputs 604 according to the current values ​​of each set of weights of one or more neural network layers 602. More specifically, the frequency prediction ML model 600 processes the set of inputs 604 through one or more neural network layers 602 to generate one or more frequency predictions 608 (described later). The frequency prediction ML model 600 processes the set of inputs 604 according to an adaptive ML algorithm and a corresponding cost function 620 that can be applied by the frequency prediction ML model 600. This is described in more detail below with reference to Figure 8.

[0085] The set of inputs 604 may include one or more indicators 606 and workload type indicators 118. Indicators 606 may include kernel-level statistics useful for accurately generating ML frequency predictions that minimize the cost function associated with the frequency prediction ML model 600. For example, these indicators 606 may include i) the number of draw calls from CPU 104 to IP block 108 representing requests to render graphic content or images, ii) the number of primitives and vertices for rendering the shapes and / or features of objects in the content / image, iii) the texture bandwidth and level of detail (LoD) for rendering the objects, iv) the texture filter type (e.g., bilinear / trilinear / anisotropic) for rendering quality relative to the angles / surfaces of the objects, and v) the resolution for object rendering (e.g., 720p, 1080p, 2k, 4k, etc.).

[0086] In some embodiments, indicator 606 includes specific performance / hardware counters or corresponding hardware signals, which provide a more precise granularity for calculating inference / predictions that enable optimization of the operating point across the entire system 100. For example, hardware indicator 606 may be performance counter values ​​and / or hardware signals that accurately indicate CPU utilization, memory utilization, and IP block utilization such as TPU / GPU / ISP / DSP utilization. Indicator 606 may include corresponding hardware counters that indicate the core processing activity of the cores of the CPU 104 or GPU 120.

[0087] In some embodiments, indicator 606 may include performance / counter signals representing a cache hit / miss indicator that reveals the amount of cache misses. For example, the cache hit / miss indicator can be processed by a DVFS prediction model 113 to predict a DVFS configuration that provides a performance boost by mitigating or minimizing cache misses that degrade system performance. Indicator 606 may also include specific performance counters for capturing load / store activity, such as the load / store ratio or the amount of memory access requests issued to memory units across the entire system 100, including memory device 128.

[0088] The generated frequency prediction 608 may include a CPU frequency prediction 610, an IP block frequency prediction 612, and a memory device frequency prediction 614. In some embodiments, indicator 606 is a kernel-level statistic specific to the GPU 120, and the IP block frequency 612 is the GPU frequency passed to the GPU 120. Similarly, indicator 606 and kernel-level statistics specific to the CPU 104 or memory device 128 can be used to generate a frequency prediction 608 that includes a CPU frequency setting passed to the CPU 104, or a DDR / DRAM frequency setting passed to the memory device 128. In some embodiments, the DDR / DRAM frequency value is passed to the memory device 128 via a memory interface (MIF) managed by the memory controller of the SoC 102.

[0089] Figure 7 shows an exemplary power prediction model 700. In some embodiments, in addition to the frequency prediction ML model 600, the power prediction ML model 700 is also included in the exemplary DVFS prediction model 113 described above with reference to Figure 1.

[0090] In the example in Figure 7, the power prediction ML model 700 can have a structure similar to, for example, substantially similar to, the frequency prediction ML model 600 in Figure 6. For example, the power prediction ML model 700 employs one or more neural networks having multiple layers of nodes that produce outputs, for example, classification, for an incoming input. In some embodiments, the neural network(s) include a hidden layer in addition to the input layer and the corresponding output layer. The neural network may be a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0091] Furthermore, some (or all) of the neural network layers of the Power Prediction ML Model 700 have a corresponding set of parameters or weights. The weights are used to process the input (e.g., a batch of inputs) through the neural network layers (e.g., hidden layers) to generate the corresponding output of the layer for computing neural network inference. In some embodiments, the weights are updated using backpropagation techniques during training of the Power Prediction ML Model 700. Similar to the Frequency Prediction ML Model 600, the kernel filters of the batch of inputs and weights processed through the layers of the Power Prediction ML Model 700 can be represented as an input tensor and a weight / parameter tensor, respectively.

[0092] The power prediction ML model 700 is configured to generate an inference output from a set of received inputs 704 according to the current values ​​of each set of weights of one or more neural network layers 702. More specifically, the power prediction ML model 700 processes the set of inputs 704 through one or more neural network layers 702 to generate one or more power predictions 708 (described below). The power predictions 708 can represent power values ​​for establishing a DVFS setting across the entire system 100. The power prediction ML model 700 processes the set of inputs 704 according to an adaptive ML algorithm and a corresponding cost function 720 that can be applied by the power prediction ML model 700. This is described in more detail below with reference to Figure 8.

[0093] The set of inputs 704 may include one or more indicators 706 and workload type indicators 118. The indicators 706 may include kernel-level statistics useful for accurately generating ML power forecasts that minimize the cost function corresponding to the power forecast ML model 700. For example, these indicators 706 may include hardware indicators, which are i) performance counters and the corresponding count values ​​generated by the performance counters (as described above in Figure 6), ii) temperature sensors and the corresponding temperature values ​​generated by the temperature sensors, iii) current sensors and the corresponding current values ​​generated by the current sensors, and iv) droop detectors and the corresponding droop values ​​generated by the droop detectors.

[0094] In some embodiments, the power prediction ML model 700 processes / analyzes indicators 706 representing temperature and / or current sensor values ​​to calculate, infer, or predict specific relationships and correlations. For example, the inference or prediction may reveal the extent to which an increase in power settings and / or frequency settings leads to a corresponding temperature rise, including any relevant correlation between current and temperature in the SoC 102 and the system 100 as a whole. In some embodiments, the temperature sensors / indicators include internal / junction temperature sensors and skin temperature sensors on a printed circuit board (PCB) that generate indications of the external temperature of the user device. The power prediction ML model 700 processes these indicators to generate power predictions 708 and relevant DVFS settings that trigger thermal throttling to adjust power consumption in the system 100. For example, the DVFS setting can dynamically set a temperature threshold or range (e.g., 85°F to 92°F) for triggering the thermal throttling function of the system 100.

[0095] In some embodiments, the droop detector is a voltage-based and / or time-based sensor that detects or measures transient droop in the supply voltage. For example, droop may occur as a result of transient current spikes caused by the switching circuits of the SoC102, and these spikes may cause localized supply voltage droop due to resistive or inductive impedances in the on-chip and off-chip power supply network. Significant or excessive voltage droop can degrade the performance and reliability of the system.

[0096] The power prediction ML model 700 can process an indicator 706, which includes at least droop detector sensor values, to generate a power setting prediction 708 that achieves an efficient power consumption level in the SoC 102. In some cases, dynamic frequency scaling or reduction may be used for droop mitigation. Thus, the power prediction ML model 700 can process the indicator 706 and determine the weights of the droop detector sensor values ​​that enable effective droop mitigation while enabling maximization of the predicted IP block frequency setting. Furthermore, the power prediction ML model 700 can generate a power setting prediction 708 that minimizes power consumption to achieve an efficient power output level for a desired operating frequency that mitigates a particular QoS violation.

[0097] The generated power level prediction 708 may include a CPU power setting 710, an IP block power setting 712, and a memory device power setting 714. In some embodiments, indicator 706 is a kernel-level statistic specific to the memory device 128, so that the predicted MIF / DDR power setting 714 includes, for example, a DRAM power value passed to the memory device 128 using the memory controller or the MIF of the SoC 102. Similarly, indicators 706 and kernel-level statistics specific to the CPU 104 or IP block 108 may be used to generate a power level prediction 608 that includes a CPU power setting passed to the CPU 104, or a power setting(possibly) of the IP block 108 passed to one or more IP devices such as the GPU 120, TPU 122, DSP 124, or ISP 126.

[0098] Figure 8 is an exemplary process flow diagram representing process 800 for training at least frequency prediction model 600 and power prediction model 700. Generally, process 800 corresponds to a model training phase that precedes the model deployment or implementation phase. During the deployment phase, the trained models are used to generate predictions / inferences that are used to adjust DVFS settings, for example, at runtime, on the SoC 102 or on IP devices coupled to the SoC 102. As described above, adjusting DVFS settings on the SoC 102 may include adjusting DVFS settings on any hardware element of the IP block 108 (e.g., an IP device). Adjusting DVFS settings

[0099] Process 800 can be implemented or executed using System 100. Therefore, a description of Process 800 can refer to the aforementioned computing resources of System 100, including the resources of SoC 102. In some examples, steps or actions of Process 800 are enabled by programmed firmware instructions, software instructions, or both. Each type of instruction may be stored in a non-temporary machine-readable storage device and can be executed by one or more processors described in this document, such as CPU 104, GPU 120, TPU 122, DSP 124, or a combination thereof, or other resources.

[0100] Process 800 includes determining and initializing one or more parameters representing an experience target value (802). In some embodiments, the experience target value is used to define or determine a QoS violation. For example, the experience target value may be a key performance indicator (KPI) for a given workload or workload type. The experience target value may be user-specific or dynamically determined in system 100. In some embodiments, the experience target value is a threshold quality control value to enforce a specific QoS requirement, for example, to ensure that an image post-processing operation is computed as quickly as possible (e.g., with minimum latency), or that a gaming application iterates through multiple scene changes without dropping a single frame or exceeding a target loss frame value.

[0101] System 100 trains the frequency prediction ML model 600 and the power prediction ML model 700, respectively (804). In some embodiments, the frequency prediction ML model 600 is trained online for a threshold period. For example, the training phase of the frequency prediction ML model 600 may take place over several hours, several days, several weeks, or several months. In some cases, the frequency prediction ML model 600 may be trained offline. The online aspect of the training phase means that the frequency prediction ML model 600 is trained on the device using violation data obtained from an exemplary violation counter on the device. For example, the violation counter may only be active when a user runs and / or interacts with the application, which corresponds to the online aspect of the training phase. The device may be the user device described above with reference to Figure 1.

[0102] System 100 can perform various weight update operations via backpropagation (806). Weight updates may include embedding layer operations that generate a set of embeddings for training the neural network. The embedding layer of the neural network is used to embed features in the feature / embedding space corresponding to the embedding layer. An embedding vector can be a vector of numbers that maps to the corresponding features in the set of features in a lookup table representing the embedding layer. Features may be attributes or characteristics shared by independent units on which analysis or prediction is performed.

[0103] For example, an independent unit could be a group of words in a vocabulary, or an image pixel that forms part of an item such as an image and other documents. An algorithm for training the embeddings in the embedding layer to map features to embedding vectors may be performed by a neural network processor. In some embodiments, the embeddings in the embedding table are learned in conjunction with other layers of the neural network on which the embeddings are used. This type of learning occurs by backpropagating gradients to update the embedding table.

[0104] Embedding outputs are generated when the neural network of system 100 is trained to perform specific computational functions, such as calculations related to machine translation, natural language understanding, ranking models, or content recommendation models. Training a neural network involves updating a set of embeddings previously stored in the neural network's embedding table, such as during the pre-training phase of the neural network. Embeddings may be trained in conjunction with the neural network in which they are used.

[0105] System 100 determines that a particular model has reached convergence (808). System 100 determines whether the frequency prediction ML model 600 has reached convergence during the training phase of the model. For example, System 100 determines the model convergence of the frequency prediction ML model 600 based on the corresponding cost function 406 of the model 600. Similarly, System 100 determines whether the power prediction ML model 700 has reached convergence during the training phase of the model. For example, System 100 determines the model convergence of the power prediction ML model 700 based on the corresponding cost function 406 of the model 700.

[0106] The DVFS prediction engine 106 can determine the extent to which a particular model reaches convergence based on parameter values ​​of a QoS cost function 406 that quantifies the error between predicted and expected values. For a given QoS cost function 406, the DVFS prediction engine 106 can generate a model convergence parameter value that characterizes the extent to which the model's predicted values ​​match or are consistent with the expected values ​​of the inference outputs generated by the model. In some embodiments, the model convergence parameter value is compared to a convergence threshold that represents a desired consistency between predicted and expected values. For example, the DVFS prediction engine 106 can determine model convergence in response to determining that a model convergence parameter value (e.g., 0.91) satisfies or exceeds a corresponding threshold (e.g., 0.90).

[0107] As described above, the adaptive learning module 112 may include multiple QoS cost functions 406, each QoS cost function 406 may correspond to one or more ML models 402. For example, the adaptive learning module 112 may be configured to access the frequency prediction ML model 600, the power prediction ML model 700, and the corresponding adaptive QoS cost function 406 for each of the workload types detectable by the frequency prediction ML model 600, the power prediction ML model 700, and / or the workload detector 110. The DVFS prediction engine 106 leverages this attribute of the adaptive learning module 112 to dynamically determine control values, including thresholds, for each QoS parameter based on one or more ML models 600, 700, workload types 118, and the corresponding QoS cost functions 406.

[0108] In response to determining that a particular model has not reached convergence, the system 100 adjusts the learning rate and the learning rate and empirical values ​​of one or more targets (810). For example, the DVFS prediction engine 106 adjusts the learning rate and the empirical values ​​of targets based on a feedback path 812. The feedback path 812 may represent or route a feedback control signal 812, such as a signal generated based on a model convergence parameter value (e.g., 0.75) that does not meet or exceed a corresponding threshold (e.g., 0.90). In some embodiments, the empirical target values ​​include at least the target values ​​described above with reference to the examples in Figures 3 and 4.

[0109] In response to determining that the model has reached convergence, system 100 performs cross-validation of its model (814). The DVFS prediction engine 106 evaluates the model's performance on novel, unseen data using exemplary cross-validation techniques. In some embodiments, system 100 analyzes or splits the system 100's available training data into various subsets, including a training dataset used to train the model and corresponding validation datasets used to evaluate the model's performance after training iterations. Each of these datasets may be described or identified as a “fold,” and a given validation dataset may represent a type of novel, unseen data for the model.

[0110] The DVFS prediction engine 106 can repeat the model training and validation process for different training iterations of a particular ML model of system 100, designating different folds or combinations of datasets as either training or validation datasets. In some embodiments, system 100 repeats this model training and validation process N times, or iterates, where N is an integer greater than or equal to 1. To evaluate the model performance in each training iteration, the DVFS prediction engine 106 reserves or designates a different fold as a validation dataset, where that particular fold is excluded from the designated training dataset.

[0111] In some embodiments, the DVFS prediction engine 106 includes a training dataset and a corresponding validation dataset for each of the ML models 402. For example, the DVFS prediction engine 106 may include or acquire these datasets for a power ML model, a frequency ML model, or both. In some examples, the DVFS prediction engine 106 may also include or acquire these datasets for ML models associated with the workload detector 110.

[0112] The DVFS prediction engine 106 can generate model performance metrics for each model's training evaluation or iteration. The DVFS prediction engine 106 can also generate a composite performance metric for a given model by combining or averaging the respective performance metrics for each model's training evaluation or iteration. System 100 performs cross-validation to ensure that the ML models selected for deployment exhibit robust performance that generalizes well to novel, unseen data. In some embodiments, the DVFS prediction engine 106 uses cross-validation to mitigate or prevent overfitting of a particular model, which can occur when the model's training is exclusively tied to observations derived from its training dataset, resulting in poor performance on novel, unseen data.

[0113] Following cross-validation of the models, system 100 determines whether the observed precision of the model meets or exceeds the target precision (816). Each of the observed precision and target precision may be characterized by a numerical value such as an integer (e.g., 7) or a decimal (e.g., 0.87). The DVFS prediction engine 106 can evaluate the precision of the model by comparing the value of the inference output generated by the model with the corresponding target value of that output. The DVFS prediction engine 106 performs this precision of the model for each of the frequency prediction ML model 600 and the power prediction ML model 700. Therefore, the examples and references for the frequency prediction ML model 600 in the following examples can also be applied to the power prediction ML model 700.

[0114] For example, during model training, for a detected workload type 118 which is a low-interaction gaming workload 228, the frequency prediction ML model 600 generates an IP block frequency prediction 612 for the GPU 120. For example, the frequency prediction 612 could be a predicted minimum GPU frequency value (e.g., 1.5 gigahertz (GHz)) that is predicted to mitigate or eliminate QoS violations associated with slow FPS values, such as lost frames and / or FPS values ​​below a certain threshold (e.g., a 60 FPS threshold). During model training, the frequency prediction 608 or power prediction 708 is a target value, and as a result, the frequency prediction 612 or power setting prediction 712 for the IP block is intended to satisfy the constraints imposed by the corresponding QoS cost functions 620, 720, respectively.

[0115] During training, the DVFS prediction engine 106 may determine that the observed accuracy of the frequency prediction ML model 600 does not meet or exceed the model's target accuracy. For example, the DVFS prediction engine 106 may make this determination if, when called upon or used to render graphic content for a low-interaction gaming workload 228, a QoS violation occurs for lost frames and / or slow FPS after the GPU 120 starts operating at the predicted minimum GPU frequency value. If the system 100 determines that the observed accuracy of the frequency prediction ML model 600 does not meet or exceed the target accuracy, the system 100 may readjust one or more empirical target values ​​associated with the model and the training stage of the model by iterating through process 800 (818), and then restart training the model in process step 804. The system 100 may make similar decisions, adjustments, and extensions of training stages for the power prediction ML model 700.

[0116] The DVFS prediction engine 106 can also evaluate model accuracy relative to target accuracy in other ways. For example, the DVFS prediction engine 106 can encode a target accuracy or frequency (or power consumption) value to access the accuracy of a frequency (or power) prediction ML model 600 (or 700). The encoded value can be derived from annotated training samples that show the tolerance range of the minimum frequency prediction value 608 for a particular workload type 110 and different combinations of input indicators 606. The DVFS prediction engine 106 can compare different frequency predictions 608 to one or more of its encoded target frequency values. Based on the results of the comparison, the DVFS prediction engine 106 determines whether the observed accuracy of the frequency prediction ML model 600 meets or exceeds the target accuracy.

[0117] System 100 terminates one or more training stages of the frequency prediction ML model 600 in response to determining that the observed accuracy of the frequency prediction ML model 600 meets or exceeds the target accuracy for the model. Relatedly, System 100 terminates one or more training stages of the power prediction ML model 700 in response to determining that the observed accuracy of the power prediction ML model 700 meets or exceeds the target accuracy for the model. For example, TPU 122 may be requested to perform intensive inference calculations for a computationally intensive ML workload 210 within a target latency of such workload type. The frequency / power prediction ML models 600 / 700 can generate frequency / power predictions 612 / 712 that include predicted minimum TPU frequency / power setpoints for running the computationally intensive ML workload 210 within the target latency.

[0118] The DVFS prediction engine 106 determines whether the frequency prediction ML model 600 meets or exceeds the target accuracy based on the predicted minimum TPU frequency value and / or the corresponding minimum power setting value. For example, the DVFS prediction engine 106 passes the frequency prediction 612 to the TPU 122, causing the TPU 122 to set its operating frequency to the minimum TPU frequency indicated by the frequency prediction 612. The TPU 122 uses this minimum TPU operating frequency to perform intensive inference calculations for the computationally intensive ML workload 210. The DVFS prediction engine 106 determines whether the TPU 122 executed the computationally intensive ML workload 210 within the target latency while the TPU 122 was operating at the predicted minimum operating frequency.

[0119] Alternatively, the DVFS prediction engine 106 determines whether a QoS violation exceeding the latency target occurred after the TPU 122 executed a computationally intensive ML workload 210 while operating at the predicted minimum TPU operating frequency. In some embodiments, the system 100 generates a corresponding QoS check in response to executing a specific workload type using predicted DVFS settings, such as predicted frequency or power values. The DVFS prediction engine 106 determines that the frequency prediction ML model 600 meets or exceeds the target accuracy if, i) the TPU 122 executes the computationally intensive ML workload 210 within the target latency while operating at the predicted minimum TPU operating frequency, or ii) no QoS violation exceeding the target latency occurs after the TPU 122 executes the computationally intensive ML workload 210.

[0120] In relation to the above example(s), the DVFS prediction engine 106 can determine that the observed accuracy of the frequency prediction ML model 600 meets or exceeds the target accuracy based on the absence of QoS violations for a given frequency prediction 608 and the corresponding workload type 118. In some embodiments, a particular frequency prediction(s) 608 is generated for a particular workload type(s) 118, and the DVFS prediction engine 106 can determine that no QoS violations occur when the frequency values(s) of the particular prediction(s) 608(s) are used to adjust the operating frequencies of the devices(s) of the system 100 used to run the particular workload type(s).

[0121] If system 100 determines that the observed accuracy of the frequency prediction ML model 600 meets or exceeds the target accuracy, system 100 deploys the DVFS prediction model 113 for use during the model implementation phase (820). The power prediction ML model 700 and the corresponding example related to the associated power prediction 708 are similarly related to process 800.

[0122] Figure 9 shows an exemplary process 900 for frequency and power prediction in a hardware integrated circuit. Process 900 can be implemented or executed using system 100 and SoC 102, which include the prediction model described above. Thus, the description of process 900 can refer to the aforementioned computing resources of system 100 and SoC 102. In some examples, the steps or actions of process 900 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction can be stored in a non-temporary machine-readable memory device and can be executed by one or more of the processors or other resources described in this document, such as CPU 104, scalar cores or compute tiles of TPU 122, hardware ML accelerators, or neural network processors.

[0123] In some embodiments, the steps of process 900 are performed by integrated hardware circuitry to generate ML outputs, which include the outputs of the neural network layers of a multilayer neural network running one or more machine learning (ML) models. For example, the outputs may be part of a computation for an ML task or inference workload to generate image processing, speech processing, or image recognition outputs. As shown above, the integrated hardware circuitry may be a dedicated neural network processor or hardware ML accelerator configured to accelerate ML computations to generate different types of data processing outputs.

[0124] Referencing process 900 again, system 100 detects that the application is running on the SoC (902). The DVFS prediction engine 106 can detect that the application is running based on data or indicator values ​​from CPU 104. For example, CPU 104 generates one or more indicators 105, such as an application launch indicator or a function call that is triggered in response to the execution or launch of the application on the user device.

[0125] System 100 determines the workload type of an application based on one or more indicators 105 that are generated concurrently with the application being run (904). The DVFS prediction engine 106 of SoC 102 computes one or more inferences based on the workload type and indicators (906). The computed inferences are used to achieve threshold quality of service ("QoS") when applications such as gaming, content streaming, or video editing are run on SoC 102.

[0126] The DVFS prediction engine 106 generates predicted frequency and power values ​​from the computed inferences (908). In some embodiments, the DVFS prediction engine 106 uses a frequency prediction ML model 600 to predict a minimum GPU frequency value (e.g., 1.5 gigahertz (GHz)) for the clock signal of the GPU 120 that minimizes QoS violations related to lost frames, FPS, or UI responsiveness for various workload types.

[0127] The predicted frequency or power values ​​are used to adjust the operating frequency or power output in the hardware integrated circuit to achieve a threshold QoS when the application runs on the SoC (910). For example, the DVFS prediction engine 106 uses its predictions to more efficiently adjust the DVFS settings to achieve the fastest frequency and lowest power consumption operating point. In some embodiments, the frequency prediction 608 and power prediction 708 provide exemplary DVFS settings for thermal throttling that are dynamically adjusted for various workload types and / or the state of external user devices.

[0128] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs encoded in a tangible non-temporary program carrier, i.e., one or more modules of computer program instructions, to be executed by a data processing device or to control the operation of a data processing device.

[0129] Alternatively or additionally, program instructions may be encoded into artificially generated propagating signals, such as mechanically generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device. Computer storage media may be machine-readable storage devices, machine-readable storage boards, random-access memory devices or serial-access memory devices, or one or more combinations thereof.

[0130] The term "computing system" encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may include dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device may also include code that constitutes the execution environment for the target computer program, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0131] Computer programs (sometimes called, or written as, programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, such as as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0132] Computer programs may, but do not necessarily, correspond to files in a file system. A program may be stored in a single file dedicated to the program itself, in a part of a file that holds one or more scripts stored in other programs or data, such as a document in a markup language, or in multiple collaborative files, such as files that store one or more modules, subprograms, or parts of code. Computer programs can be deployed to run on a single computer, or on multiple computers located in one location or distributed across multiple locations and interconnected by a communication network.

[0133] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. The processes and logic flows can also be performed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units), and these devices can also be implemented as dedicated logic circuits.

[0134] A computer suitable for running computer programs may include, for example, a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit, and may be based on them. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Some elements of a computer are the central processing unit for executing and running instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, including, for example, magnetic disks, magneto-optical disks, or optical disks, or may be operably connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, computers may be incorporated into other devices, such as, for example, mobile phones, personal digital assistants (PDAs), mobile audio players or mobile video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives.

[0135] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, which include, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0136] To provide user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device for displaying information to the user, such as an LCD (liquid crystal screen) monitor, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. Other types of devices can also be used to interact with the user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and the input from the user may be received in any form, including acoustic, voice, or haptic input. In addition, the computer can interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0137] Embodiments of the subject matter described herein can be implemented in a computing system that includes a backend component, e.g., a data server, or a middleware component, e.g., an application server, or a frontend component, e.g., a client computer having a graphical user interface or web browser on which a user can interact with embodiments of the subject matter described herein, or one or more such backend components, middleware components, or frontend components in any combination thereof. The components of the system can be interconnected by digital data communications, e.g., any form or medium of a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0138] A computing system can include clients and servers. Clients and servers are generally remote to each other and typically communicate through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other.

[0139] While this specification includes details of many specific embodiments, these should not be interpreted as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a secondary combination or a variation of a secondary combination.

[0140] Similarly, while the drawings show operations in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0141] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying drawings does not necessarily require to be performed in the specific order or sequence shown to obtain the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for predicting frequency and power in a hardware integrated circuit, The process involves detecting that the application is running on the system-on-a-chip ("SoC") of the aforementioned hardware integrated circuit, The workload type of the application is determined based on indicators generated simultaneously with the application being executed, Based on the workload type and the indicators, the SoC's Dynamic Variable Frequency Signal ("DVFS") prediction engine computes inferences used to achieve threshold quality of service ("QoS") when the application runs on the SoC, The DVFS prediction engine generates predicted frequency values ​​and predicted power values ​​from the calculated inferences, A method comprising: adjusting the operating frequency or power output of the hardware integrated circuit to achieve a threshold QoS when the application is run on the SoC, using the predicted frequency value or the predicted power value.

2. Generating the predicted frequency value and predicted power value from the calculated inference is: The method according to claim 1, comprising generating a minimum frequency value and a corresponding minimum power value by the DVFS prediction engine in order to achieve the threshold QoS when the application is executed on the SoC.

3. Generating the predicted frequency value and predicted power value from the calculated inference is: The method according to claim 1, comprising generating a maximum frequency value and a corresponding minimum power value required to implement the maximum operating frequency in the hardware integrated circuit using the DVFS prediction engine based on the maximum frequency value.

4. Calculating the aforementioned inference is, The inference is calculated by the frequency prediction model of the DVFS prediction engine based on the workload type and the indicator. The power prediction model of the DVFS prediction engine calculates inference based on the workload type and the indicator, The method according to claim 1, including the method described in claim 1.

5. The aforementioned SoC is integrated into the user device, The method according to claim 1, wherein the frequency prediction model is trained and executed on the SoC.

6. The aforementioned power prediction model is a pre-trained neural network model. The method according to claim 5, wherein the power prediction model is trained offline, isolated from the SoC and the user device, and then loaded onto the SoC for execution on the user device.

7. The SoC is for a user device, and the method further, Determining a first workload type for a first application running on the user device, Based on the first workload type, the hardware integrated circuit adjusts the operating frequency and power output using predicted frequency and power values ​​generated from inferences calculated by the DVFS prediction engine. Determining the second workload type of the second application running on the user device, Using predicted frequency and power values ​​generated from inferences calculated by the DVFS prediction engine based on the second workload type, the hardware integrated circuit adjusts the operating frequency and power output. The method according to claim 1, including the method described in claim 1.

8. To achieve a first threshold QoS when the first application is executed on the SoC, the operating frequency or power output is adjusted by referring to the first workload type, To achieve a second threshold QoS when the second application is executed on the SoC, the operating frequency or power output is adjusted by referring to the second workload type, The method according to claim 7, further comprising:

9. The first workload type and the second workload type are different workload types. The method according to claim 8, wherein the first application and the second application are the same application.

10. Calculating the aforementioned inference is, The method according to claim 1, comprising calculating the inference based on the workload type, a plurality of software indicators, and a plurality of hardware indicators.

11. A system-on-a-chip ("SoC") that performs frequency and power prediction for a hardware integrated circuit, wherein the system-on-a-chip is Processing device and The system includes a non-temporary machine-readable storage medium for storing instructions that can be executed by the processing device in order to bring an operation to be performed, The aforementioned operation is, Detecting that the application is running on the aforementioned hardware integrated circuit, The workload type of the application is determined based on indicators generated simultaneously with the application being executed, Based on the workload type and the indicators, the SoC's Dynamic Variable Frequency Signal ("DVFS") prediction engine computes inferences used to achieve threshold quality of service ("QoS") when the application runs on the SoC, The DVFS prediction engine generates predicted frequency values ​​and predicted power values ​​from the calculated inferences, Using the predicted frequency value or the predicted power value, the hardware integrated circuit adjusts the operating frequency or power output to achieve a threshold QoS when the application is executed on the SoC, A system-on-a-chip that includes this.

12. Generating the predicted frequency value and predicted power value from the calculated inference is: The SoC according to claim 11, comprising generating a minimum frequency value and a corresponding minimum power value by the DVFS prediction engine in order to achieve the threshold QoS when the application is executed on the SoC.

13. Generating the predicted frequency value and predicted power value from the calculated inference is: The SoC according to claim 11, comprising generating a maximum frequency value and a corresponding minimum power value required to implement the maximum operating frequency in the hardware integrated circuit by the DVFS prediction engine based on the maximum frequency value.

14. Calculating the aforementioned inference is, The inference is calculated by the frequency prediction model of the DVFS prediction engine based on the workload type and the indicator. The power prediction model of the DVFS prediction engine calculates inference based on the workload type and the indicator, The SoC according to claim 11, including the following:

15. The aforementioned SoC is integrated into the user device, The SoC according to claim 11, wherein the frequency prediction model is trained and executed on the user device.

16. The aforementioned power prediction model is a pre-trained neural network model. The SoC according to claim 15, wherein the power prediction model is trained offline separately from the SoC and the user device, and is then loaded onto the SoC for execution on the user device.

17. The SoC is for a user device, and the method further, Determining a first workload type for a first application running on the user device, The hardware integrated circuit adjusts the operating frequency and power output using predicted frequency and power values ​​generated from inferences calculated by the DVFS prediction engine based on the first workload type, Determining the second workload type of the second application running on the user device, The hardware integrated circuit adjusts the operating frequency and power output using predicted frequency and power values ​​generated from inferences calculated by the DVFS prediction engine based on the second workload type, The SoC according to claim 11, including the following:

18. The aforementioned operation further, To achieve a first threshold QoS when the first application is executed on the SoC, the operating frequency or power output is adjusted by referring to the first workload type, To achieve a second threshold QoS when the second application is executed on the SoC, the operating frequency or power output is adjusted by referring to the second workload type, The SoC according to claim 17, including the following:

19. The first workload type and the second workload type are different workload types. The SoC according to claim 18, wherein the first application and the second application are the same application.

20. Calculating the aforementioned inference is, The SoC according to claim 11, comprising calculating the inference based on the workload type, a plurality of software indicators, and a plurality of hardware indicators.

Images