Technology to efficiently operate processing systems based on the energy characteristics of instructions and machine learning

Abstract

The present invention discloses a technique for efficiently operating a processing system based on the energy characteristics of instructions and machine learning. An integrated circuit, such as a graphics processing unit (GPU), includes a dynamic power controller for adjusting an operating voltage and / or an operating frequency. The controller can receive a current power used by the integrated circuit and a predicted power determined based on instructions to be processed in multiple processors. The controller determines the adjustment required to the operating voltage and / or the operating frequency to minimize the difference between the current power and the predicted power. An in-system reinforcement learning mechanism is included to self-adjust the controller parameters.

Description

[0001] Priority claim

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 716,513, filed on August 9, 2018, entitled “Energy aware instruction based power management for efficient system operation,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This technology relates to integrated circuit power management, and more particularly to dynamic voltage and / or frequency adjustment for power management in integrated circuits. More particularly, this technology relates to managing power based on sensing upcoming processor instructions and employing in-system enhanced learning mechanisms to predict / improve power control system parameters. Use cases include power management of massively parallel processors such as GPUs. Background Art

[0004] Graphics processing units (GPUs) have become ubiquitous. No longer used just for graphics, GPUs are now used in a wide range of applications that benefit from intensive computational operations, including, for example, artificial intelligence, real-time pattern recognition, and autonomous vehicle control, among countless other applications.

[0005] Many GPUs are massively parallel, meaning they contain many compute elements (e.g., programmable streaming multiprocessors (“SMs”)) that operate in parallel. This massively parallel architecture allows developers to break complex computations into smaller parallel parts that will complete faster because they are executed simultaneously. While very fast, this array of parallel compute elements consumes a lot of power and generates a lot of heat. As a result, power management has become an important aspect of the design and operation of GPUs (and other complex integrated circuits).

[0006] Just as a dancer uses more energy when dancing fast than when dancing slowly, digital circuits generally use more power when they run fast. In most integrated circuits, the speed of operation is controlled by a clock circuit. The clock circuit sends a "beat" (clock signal) to the various circuits on the chip. These circuits synchronize their operations to the "beat" of the metronome, which is like a clock signal. The faster the beat, the faster the circuit runs and the more power it consumes. Also, the faster the beat, the more "moves" the circuit can perform in a given time.

[0007] Supply voltage also affects speed and power consumption. Transistors (e.g., MOSFETs) are used to build digital logic circuits, which are an integral part of modern high-speed processors. When the supply voltage is increased, the transistors operate (switch) faster. One way to think about this is that switching a MOSFET transistor involves charging and discharging a capacitor. Increasing the DC voltage supplying the MOSFET can charge the MOSFET's capacitor faster, thereby increasing the switching speed of the transistor. This is similar to using a higher water pressure to fill a bucket faster. The fastest speed at which the MOSFET can be clocked can therefore depend on the supply voltage.

[0008] However, running a circuit at a higher voltage increases its potential operating speed while causing the circuit to consume more power. Generally speaking, the more power a circuit consumes, the more heat it generates. While fans and heat sinks are often used to remove excess heat, improper heating wastes power that could otherwise be used for computing. For mobile applications such as unmanned vehicles and portable devices, wasted power can unnecessarily shorten battery life. Even for stationary applications such as servers and desktop computers, wasted energy can be bad for the environment and increase operating costs.

[0009] While the most straightforward solution is to slow down the circuit to save power, this will adversely affect the number of operations per second the circuit can perform, and therefore the speed at which it can perform complex calculations. In many demanding applications, including but not limited to unmanned vehicles, user-interactive servers, and other computing devices, high-speed performance is important for safety-critical calculations and other calculations that must be performed in real time or near real time.

[0010] Therefore, there is a trade-off between operating speed and power consumption. To increase the number of operations per second, the rate (frequency) of the clock signal can be increased (and the power supply voltage can also be increased if required to support the higher clock rate), at the expense of additional power consumption. To save power, the clock signal rate can be reduced (and the power supply voltage can also be reduced if necessary), at the expense of reduced processing speed.

[0011] To manage this tradeoff, some GPUs and other processors provide dynamic control of clock rates (and in some cases, dynamic control of supply voltage to support higher clock rates) based on computational load, allowing circuits to "slow dance" when processing demands are low and allowing them to "fast dance" when more computational speed is needed. This dynamic control can maximize speed performance while reducing overall power consumption and corresponding thermal output.

[0012] While much work has been done in the past, further improved solutions are needed to provide an adaptive yet adjustable system that can handle sudden load steps and load releases without compromising overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The following detailed description of exemplary, non-limiting illustrative embodiments is to be read in conjunction with the accompanying drawings, in which:

[0014] Figure 1 and Figure 2 An example system including active power management control in accordance with certain example embodiments is schematically illustrated.

[0015] Figure 3 A flow chart illustrating a power control technique using current and predicted GPU power according to certain example embodiments.

[0016] Figure 4 and Figure 5 A circuit for implementing active power management techniques using a learning mechanism according to certain example embodiments is schematically illustrated.

[0017] Figure 6 Schematic diagram of a reinforcement learning proportional-integral-derivative controller.

[0018] Figure 7 Schematic diagram of a scalable reinforcement learning proportional-integral-derivative system.

[0019] Figure 8 is a schematic diagram of a scalable reinforcement learning proportional-integral-derivative system including two PID controllers and two learning systems.

[0020] Fig. 9 Reinforcement learning feedback control methods according to certain embodiments are described.

[0021] Detailed Description of Example Embodiments

[0022] Many GPUs use DVFS (Dynamic Voltage and Frequency Scaling) when running, which deploys load-based voltage and frequency settings of the GPU to obtain the highest performance within the power envelope. In this case, load is defined as the energy intensity of the work being done on the GPU. Since math operations running on the streaming multiprocessors (SMs) within the GPU consume high energy per clock cycle, math-dominated workloads constitute a large load. Conversely, when the work is memory-dominated and not as much math operations are used in the SM, the work is considered to be medium or low load (in this case, the SM may spend a lot of time waiting for the memory system to retrieve data from memory).

[0023] Essentially, load-based DVFS is reactive in nature. However, this load-based DVFS may lead to undesirable behavior where the operating point at high voltage / high frequency may suddenly experience a load step on the GPU. This is mainly because the work launched on the GPU can quickly switch from a math-dominated regime to a memory-dominated regime or vice versa during the short timescale when DVFS cannot operate. A load step is when a high-energy operation may occur because the energy consumed in the GPU increases at a given voltage and frequency. For the GPU, this may occur when it executes high-power math instructions. This situation can cause peak power overshoots for a short period of time, which in turn stresses the GPU's power / energy delivery system and may cause power outages or reliability issues.

[0024] High energy instructions (e.g., Matrix Math Instructions (MMA) or Tensor Core instructions, which allow for the acceleration of large matrix operations at the heart of AI) can significantly increase the energy consumed in a clock cycle and can have an impact on the power delivery network, causing reliability or timing issues. This results in the GPU having to maintain the performance limit consumed. To address these issues, we need an adaptive but adjustable system to ensure that we can adjust energy consumption to mitigate these adverse effects without affecting overall performance. Fundamentally, current methods for dealing with this effect are not sufficient to achieve the optimal operating point management described below.

[0025] Current methods for preventing short-term power excursions include:

[0026] 1) Preventive - limiting the maximum operating frequency to prevent or limit such excursions; and

[0027] 2) Reactive - actuated by deploying sensors that sense the increase in power / current and then reduce the operating voltage and frequency through adaptive voltage and frequency scaling.

[0028] These approaches can be effective but have their caveats / limitations.

[0029] The first approach limits the maximum frequency at which the GPU can reliably run and does not achieve optimal performance. This approach ensures by design that peak power is limited for short periods of time, but capping the maximum operating frequency may limit performance over longer periods of time that are important for measuring the energy efficiency of the GPU.

[0030] The second method is a reactive method and depends on the time of sensing and activation where there may be a delay, so they are effective but the period of effectiveness does not usually extend below 100usec.

[0031] This is because measuring changes in load in a reactive manner has a cost, as changes in load appear as changes in power / current draw, which have a sensing delay associated with them. When the system designer attempts to reduce the period of peak power management, she encounters some external component limitations (e.g., channel multiplexing) that may be a bottleneck to speeding up sensing.

[0032] Another approach is to hint at changes in workload (load step / release) in the field so that this proactive approach can guide power management to adjust delivery to prevent reliability issues. However, detecting sudden and large changes in workload is particularly difficult in the front-end stages of the GPU pipeline, especially for compute workloads (where program semantics are not visible in the front-end of the GPU pipeline). However, the later stages of the GPU pipeline (e.g., in the streaming multiprocessors (SMs) of NVIDIA GPUs) are ideal locations for detection because the semantics are visible as instructions in the program can be decoded to identify such changes in the workload.

[0033] Load change management is not limited to load steps (i.e. increase in current / power), but also requires management of sudden decreases in power (load release). Sudden load releases can cause supply overshoots due to stored energy in reactive components, which can cause inductive kickback and can also cause the power conditioning system to trip.

[0034] Detecting sudden and large changes in workload is particularly difficult in the front-end stages of the GPU pipeline, but the intermediate stages of the GPU (such as the SM in NVIDIA GPUs) can detect instructions in the program that may effectively cause load step / release behavior before the instructions are executed.

[0035] Instruction pre-decoding to estimate future power requirements

[0036] Example embodiments provide an active sensing method by which potential or possible load step events (e.g., by SM) may be identified through instruction pre-decoding. Instruction pre-decoding by an instruction look-ahead pre-decoder, which operates on a queue or buffer of fetched instructions prior to executing the instructions, allows instructions to be marked / identified prior to scheduling them for execution. Such an instruction look-ahead pre-decoder may be in addition to a typical instruction decoder for decoding instructions for execution. Instruction look-ahead pre-decoding allows notification of an impending load step / release event several clock cycles before the event actually occurs. Notification of this event to a power management unit in advance allows changes to be made to prevent peak power overshoot conditions.

[0037] The power management unit tracks the operating voltage and frequency and can use this notification to take action to move the voltage and / or frequency toward the operating regime to prevent the high power event. Moving the voltage and / or frequency in this way to reduce power can reduce electrical reliability issues for shorter periods of time without affecting performance.

[0038] From the architectural analysis, example embodiments can clearly identify instructions that, when executed, will result in high energy consumption within a clock cycle, which will result in higher power. Instruction pre-decoding can classify these instructions as they pass through, for example, the front end of the SM pipeline and notify the GPU-level power management unit (PMU) of these events. Once high-energy instructions are identified (e.g., in the SM), information is consumed to initiate hardware and / or software changes to manage peak power steps or releases.

[0039] At the GPU level, the PMU aggregates these notifications from, for example, all SMs in a GPU, and then based on the temporal and spatial characteristics of these events, can calculate the likelihood of the magnitude of a sudden work increase or decrease. This allows the PMU to make proportional changes based on the full GPU workload, as the workload running on each SM may vary. The PMU makes changes over appropriate periods to only implement clock frequency or voltage / frequency changes to limit peak power where high power events are expected, thereby avoiding overshoot and improving system-level operation.

[0040] In one non-limiting embodiment, each SM includes a pre-decoder that outputs a digital code that is encoded to represent the energy of instructions to be executed in the future. The digital codes from each SM are accumulated, filtered, and smoothed by digital processing to be fed to the control system. Since the use of an offline supervised training system to derive the parameters of the control system is limited, the example embodiment uses an in-system learning mechanism to continuously improve the parameters of the control system. The example embodiment allows for improved performance within a power envelope (Perf / W) using robust system-level operation in a short period of time; for example, in a server-based system where it is desired to have the highest efficiency (performance at the power envelope) and high system operation reliability.

[0041] Actuation

[0042] In an example non-limiting embodiment, sensing high energy instruction changes from the SM is used to actuate changes in hardware or software to manage peak power steps or releases. At the GPU level, the PMU aggregates these notifications from all SMs in the GPU and then, based on the temporal and spatial characteristics of these events, can calculate the likelihood of the magnitude of the sudden work increase or decrease.

[0043] One approach according to an example embodiment is to use a "PID" (Proportional-Integral-Derivative) controller, which allows for fixed or self-adjusting parameters of the controller. The PID controller output can perform frequency-only, voltage-only, or voltage / frequency changes based on the period of time the power is managed.

[0044] In an example non-limiting embodiment, high energy instruction notifications from the SM are aggregated into an IIR (infinite impulse response) filter, the output of which provides an effective change in the computational energy of the GPU. The computational energy and the set voltage and frequency give a near-term power estimate. The example embodiment takes the difference between this future power estimate and the current measured power. This difference, that is, the error or increment, is fed into the PID controller. The example embodiment provides an adaptive PID controller in which the PID control parameters are not static, but are updated based on a deep reinforcement (reinforcement) learning method. The period of the machine learning method can be different from the period of operation of the PID, allowing, for example, the PID to run much faster than the adaptation speed provided by the learning method.

[0045] Because workloads can be diverse and change over short periods of time, some example non-limiting embodiments use a reinforcement learning approach that uses Actor-Critic or Q Machine Learning (ML) to update PID parameters. Reinforcement learning can observe the impact of specific operating parameters of the PID on peak power steps and releases, and adjust these operating parameters to optimize power management performance.

[0046] In an example non-limiting embodiment, a PID controller moves the frequency and / or voltage to reduce the error to zero (or at least minimize it), and feeds the inputs (error, measured power, and predicted power) in parallel into a PMU processor or controller. The PMU processor / controller runs a self-tuning learning algorithm that can periodically update controller parameters based on, for example, characteristics of the error.

[0047] The hardware (HW) actuated and software (SW) adjusted methods allow for advanced control systems that predict power excesses to implement frequency-only or voltage / frequency changes to limit peak power when high power events are expected, thereby improving system-level operation and reliability when operating at maximum efficiency (Perf@W). As the enhanced machine learning gains more experience with power management results on the specific GPU it operates on, it more effectively adjusts performance to avoid over-power (step and release) scenarios that the PID controller would otherwise issue commands based on expected computational load. In some embodiments, the enhanced machine learning is able to command the SM to change its instruction execution (e.g., by delaying execution of high power instructions or load balancing the power required to execute instructions between different time intervals) to further avoid system over-stepping and over-release.

[0048] High-level view of power management controls

[0049] Figure 1 and Figure 2 An example system 100 is schematically illustrated, including active predictive, eg, machine learning based, power management control, in accordance with certain example embodiments.

[0050] System 100 may include one or more GPUs 102(0), ... 102(N) communicating with each other via one or more PCI or other digital communication bus complexes. GPU 102 may be connected to associated memory 104(0), ... 104(N). In some embodiments, system 100 may include one or more CPUs (not shown) or other control processors communicating with one or more GPUs 102 via the digital communication bus complex, and the control processors and GPUs may access respective associated memory and / or unified memory.

[0051] GPU 102 may be any type of GPU, such as an architecture having a large number of parallel processors 116. For example, each GPU 102 may include a processing complex comprised of a large number of streaming multiprocessors (SMs) 116(0), ... 116(M) configured for highly parallel operations.

[0052] Figure 1 A system 100 is illustrated, which shows that a GPU 102 is powered by a power distribution network 110. A power management unit ("PMU" or "power management") 140 monitors the operating voltage and / or current of the power provided to the GPU by the power distribution network 110 and / or the operating frequency of the GPU 102. The power management unit 140 also receives a signal 142 from one or more parallel processors 116 indicating the power required to execute instructions that have not yet been executed. For example, the processor 116, in conjunction with the power management 140, can determine how much power the processor will require to execute pending or future instructions by instruction lookahead pre-decoding before dispatching the instructions (e.g., by a scheduler or similar structure) for execution by the processor. Based on the current operating voltage and / or frequency of the GPU 102 and the signal from the processor 116, the power management unit 140 predicts or estimates the amount of power that the processor will require in the near future and controls the voltage provided to the GPU and / or the operating frequency of the GPU clock signal so that when the instructions are executed by the processor 116, the appropriate power level is provided to the GPU 102.

[0053] As an example, when a math-dominated workload that consumes high energy in a clock cycle is about to be scheduled for execution by the processor 116, the power management 140 will receive signaling from the processor 116 indicating that they will execute the math-dominated workload. In response, the power management 140 predicts that the processor 116 will require higher power in the next few clock cycles (i.e., a few cycles later than the cycle currently executed by the processor). Based on the predicted power and the current power drawn by the GPU 102, the power management 140 can change the operating frequency and / or voltage provided to the GPU 102 so that the GPU runs at an optimal power (not too high and not too low) to execute instructions quickly and efficiently. When a memory-dominated workload that consumes lower energy in a clock cycle is about to be scheduled for execution by the processor 116, the power management 140 will receive a signal indicating that instructions requiring lower power will be executed in a few clock cycles. Based on the predicted power and the current power drawn by the GPU 102, the power management 140 can reduce the operating frequency and / or voltage provided to the GPU 102 so that the GPU can still effectively execute the memory-dominated workload without using more power than necessary.

[0054] In some exemplary embodiments, when instruction pre-decoding alerts power management 140 that many high-power instructions (such as tensor or other intensive computational instructions) are about to be executed by several different processors 116, and power management 140 predicts that such execution may result in unacceptably high peak power consumption (e.g., due to overshoot), power management may reduce clock frequency, power supply voltage, or both voltage and frequency to limit power when a high-power event is predicted. By limiting power by reducing frequency and / or voltage when a high-power event is predicted, overcurrent conditions can be avoided, which may result in circuit damage, power supply fluctuations, or overheating.

[0055] like Figure 1 As shown, power management 140 monitors signal 114 of current and / or voltage sampling information received from signal conditioner and multiplexer (MUX) 112, which samples one or more power rails of power distribution network 110. Signal 114 indicates the current power provided to GPU 102. In some embodiments, GPU 102 can select a specific power rail to monitor via signal 115. Power management 140 also receives signal 142 from one or more parallel processors 116. GPU 102 controls the power it receives by controlling power distribution network 110 via signal 125 based on the difference between the current power represented by signal 115 and the predicted power determined based on signal 142.

[0056] As will be discussed in more detail below, power management 140 may include self-learning and self-adjustment of parameters for predicting and setting operating voltage and / or frequency settings. In some embodiments, self-learning may be performed in part by a bundle of parallel processors 116 using one or more deep neural networks accelerated by a hardware-based deep learning accelerator (DLA) 141 included as part of GPU 102.

[0057] In some embodiments, power management 140 may be disposed on the same substrate or die as parallel processor 116. In some embodiments, one or more components of power management 140 may be provided on a different substrate or die than the substrate or die on which parallel processor 116 is disposed.

[0058] According to an example embodiment, system 100 may be a board including one or more GPUs, one or more control processors (e.g., CPUs), and associated memory and / or memory management circuits. In some embodiments, GPU 102 may be a system on chip (SoC) including a GPU. In yet other embodiments, instead of GPU 102, system 100 may include an integrated circuit with another type of processor, a frequency voltage control system, and an ADC located on the same or different substrate. The embodiments of the present application are not limited to supplying power to a GPU and monitoring the power used by the GPU in a graphical manner. Other embodiments known to those of ordinary skill in the art may be used to supply power to a GPU and / or monitor the power used by the GPU.

[0059] More detailed schematic block diagram

[0060] Figure 2 System 100 is described in greater detail, showing example components of power distribution network 110 and power management 140 provided on GPU 102. GPU 102 includes a machine learning based system (e.g., including one or more neural networks) 150 configured to monitor the power being used by the GPU, predict the power required by GPU 102 to execute certain upcoming instructions, and control voltage and / or frequency power settings of power control system 122.

[0061] As discussed above, the power consumed by GPU 102 may vary based on the amount of processing work that is (to be) dispatched for processing by processor 116. For example, when all (or most) of a large number of parallel processors 116 (e.g., SMs) of GPU 102 are busy processing a computational instruction stream at a high supply voltage and clock rate, GPU 102 may draw a large amount of current from the board and / or through voltage regulator 105. In contrast, when processing a memory-dominant instruction stream, GPU 102 may draw significantly less power while running at the same clock rate because, for example, its many parallel processors 116 may be idle most of the time due to latency in memory accesses. Like a food server waiting for halftime at a concession stand at a football stadium, processor 116 may have little to do except wait while GPU 102 accesses memory 170. In an example embodiment, memory-dominant instructions may be processed at a lower supply voltage and clock speed, thereby reducing power consumption and heat generation. On the other hand, when GPU 102 is commanded (e.g., by a CPU, not shown) to perform complex computations on data retrieved from memory, the GPU's streaming multiprocessors 116 (like the hot dog stand staff at a football stadium when halftime starts and hungry fans have only a few minutes to buy a hot dog or pretzel and return to their seats before the second half begins) are working as fast as possible to complete their tasks.

[0062] In the disclosed embodiment, GPU 102 includes dynamic voltage / frequency scaling (DVFS) to scale the voltage provided to the GPU and / or the frequency of the GPU's internal core clock based on the current power drawn by GPU 102 and the predicted power determined based on instructions to be executed in the near future by parallel processors 116. By way of a rough analogy, the example embodiment looks at a football field and a crowd and attempts to predict when halftime traffic will begin, how long the line at the concession stand will be, and what a customer is likely to order when they get to the front of the line. This prediction allows the concession stand staff to prepare a sufficient number of hot dogs and pretzels in a timely manner without preparing too many.

[0063] power supply

[0064] In more detail, Figure 2One or more power supply units (PSUs) 108 are shown supplying power directly to various parts of the system 100, and may also be powered via power rails or connections provided by the PCI bus 106. For example, one or more GPUs 102 are powered primarily or entirely by power obtained by a voltage regulator 105 directly from the PSU 108 and / or via the PCI bus 106 power rail. The voltage regulator 105 gradually steps down the power supply voltage from the PCI bus 106 level and / or the PSU 108 level to one or more voltage levels (programmable) required to operate the GPU 102, and provides the stepped-down voltage to the GPU 102 via the GPU power rail 127. In some embodiments, the PSU 108 may alternatively or additionally provide power directly to the GPU 102 via the power rail 111.

[0065] In some embodiments, a voltage regulator (VR) 105 performs a DC to DC conversion from a high voltage to a low voltage. This conversion is used in some example embodiments because the processing chip (e.g., GPU 102) is designed to operate at a lower than standard 12VDC or 5VDC supply voltage (e.g., 1VDC-2VDC), which is a typical voltage of the PCI or PCIe bus 106 and / or the power supply unit 108 distributed on the power distribution network of the power distribution board. The voltage regulator 105 is also designed to regulate the DC output voltage it provides to the GPU. The output 127 of the voltage regulator provides current to the GPU power rail 127 for distribution to various parts of the GPU 102 and its associated supporting components and circuits.

[0066] As described above, multiple input power rails provide 127 power to GPU 102. Thus, although not specifically illustrated, voltage regulator 105 may include separate instances of each rail. That is, in some embodiments, voltage regulator 105 may include multiple voltage regulators, each of which may be individually and / or independently controlled to set a respective reference voltage, etc., for each of the multiple power rails of GPU 102.

[0067] Power Monitor

[0068] GPU 102 may include an analog-to-digital converter (ADC) 118 for monitoring (e.g., digitally sampling) a “power monitor in” analog signal 114. For example, analog signal 114 may represent current and / or voltage levels provided on board input power rail 111 and GPU power rail 127.

[0069] GPU 102 controls analog multiplexer (MUX) 112 using interface 115 to select a particular one of the multiple power rails to be sampled for a particular time division multiplexed (TDM) time slot. Thus, in some example embodiments, GPU 102 may control MUX 112 to select a "round robin" sequence of different analog signal levels, allowing the GPU to periodically monitor various power signal levels, including the power level currently being supplied to GPU 102 itself. In other embodiments, GPU 102 controls MUX 112 to select inputs to be monitored as desired.

[0070] Thus, the signal conditioning and MUX circuit 112 selects between multiple input power rails 127 for powering the GPU 102 and multiplexes the selected rails to the ADC 118 via the MUX-ADC interface 114, with the selection of which signal to supply to the GPU at a particular time being made by the GPU itself in an exemplary non-limiting embodiment. The circuit 112 may also perform signal conditioning on the signals provided to the GPU. For example, the circuit 112 may use a voltage divider or other circuit to scale voltage levels and / or convert current levels to voltage levels representative of current levels. Thus, in some embodiments, the "power monitor input" signal 114 may include (1) an analog voltage whose level represents or indicates the amount of current being drawn by the GPU 102; and (2) an analog voltage (which may be scaled down) representing the voltage level being supplied to the GPU in response to different selections of the MUX 114. Based on these two values, the GPU 102 may determine the amount of power currently being consumed (in Watts).

[0071] The external signal conditioning and MUX circuit 112 can use a single "power monitor input" analog input to the GPU 102 and ADC 118 to monitor multiple input levels by multiplexing between multiple input power rails 127, thereby reducing the I / O pin count of the GPU 102. The MUX-ADC interface 114 can be very sensitive to the details of the MUX 112 and ADC 118. The interface 114 is configured with MUXs and ADCs that are matched to each other in terms of impedance, signal range, etc. However, other arrangements can be used. For example, multiple ADCs 118 can be provided for the GPU 102 to enable it to handle multiple power monitoring inputs simultaneously.

[0072] In one example embodiment, analog-to-digital converter (ADC) 118 may be configured to monitor the power supply voltage and current levels currently provided to GPU 102. Figure 2As shown, in one embodiment, an off-chip analog multiplexer (MUX) 112 receives as input the levels of supply voltage and current used to power GPU 102. Signal conditioning, such as a voltage divider / scaler circuit in MUX 112, provides signals representing the voltage levels and current levels provided by PSU 108 directly or through voltage regulator 105. GPU 102 controls MUX 112 via signal 115 to select a specific level signal to provide to the GPU's "monitor in" input. U.S. Patent Application 16 / 046,916 (filed on July 31, 2018, entitled "Voltage / Frequency Scaling for Overcurrent Protection with On-Chip ADC") discloses additional details of using an on-chip ADC to monitor GPU power and control the power provided to the GPU.

[0073] In another embodiment, ADC 118 may monitor the “power in” provided by voltage regulator 105 to determine the current power used by GPU 102 .

[0074] Processor pre-decoding and aggregation

[0075] The processor 116 is configured to generate a signal indicating the power level required for the instruction to be executed based on the instruction type. The processor 116 generates such a signal for the instructions to be executed before executing these instructions. Each processor 116 can (e.g., partially) decode each pending instruction (e.g., opcode) in a given cycle to determine whether the instruction is a low-power instruction or a high-power instruction. In one embodiment, each processor 116 can generate a binary signal or flag to indicate whether the respective processor will use a high level or a low level of power (e.g., a flag is set when the processor unit will execute an instruction that requires a high level of power). In other embodiments, the signal indicating the expected or future power usage in a given cycle can be represented by a range (e.g., a value between 0 and n, where n is an integer). The processor 116 can continuously parse the instructions received for execution in the near future and provide a power estimate for executing the pending instructions several cycles before execution.

[0076] In one example embodiment, filter / scaler 420 receives an aggregate of power estimation signals from complex processor 116, filters the aggregate to remove certain types of noise, and scales the result. Filter / scaler 420 combines or compares the scaled result with the power level currently being used by the GPU sensed / detected by ADC 118. The result of the combination or comparison forms an error or delta signal that is used to change a control signal that controls frequency-voltage controller 122.

[0077] Thus, power management 140 receives a signal from ADC 118 indicating the current power used by GPU 102 and a signal indicating the estimated future power requirements of processor 116, and determines whether to change the operating voltage and / or frequency. More specifically, power management 140 may include circuitry such as filter / scaler 420 to accumulate signals from processor 116, estimate the power requirements of GPU 102 in a given future cycle, determine the error between the estimated power of GPU 102 and the current power used by GPU 102, and output control signals to change the operating frequency and / or voltage in time to meet the changed power requirements. For example, if the error is below a certain threshold indicating that the power requirements have not changed much, power management 140 may maintain the current settings.

[0078] Frequency and / or voltage controllers and learning systems

[0079] As will be discussed in more detail below, the frequency / voltage control system 122 may include a proportional-integral-derivative (PID) controller configured to receive a signal indicative of an error and output a signal for moving the frequency and / or voltage up or down to reduce the error to zero. The learning system 150, which may be implemented in software and / or hardware, may be configured to monitor the operation of the frequency / voltage control system 122 and update parameters (e.g., constants or coefficients used by the PID controller) used to determine output signals to improve the performance of the frequency / voltage control system 122.

[0080] For example, the learning system 150 may modify the control parameters of the frequency / voltage control system 122 to reduce errors more quickly and / or reduce overshoot when moving the frequency and / or voltage. The learning system 150 may receive the error, the current power used by the GPU 102, and / or the predicted power of the processor 116, and run a self-tuning learning algorithm that modifies the parameters used by the frequency / voltage control system 122. The learning system 150 may monitor the operation of the frequency / voltage control system 122 for a predetermined number of cycles and periodically update the parameters used by the frequency-voltage control system 122.

[0081] The learning system 150 may run on a different (e.g., slower) period of time than the frequency-voltage controller 122, and in some cases may include software running on one or more processors 116 accelerated by the deep learning accelerator 141. In such an embodiment, the learning system 150 is used in part to predict the power that the processor 116 will require to execute the learning system itself.

[0082] The output control signals of the frequency / voltage control system 122 may control at least one voltage regulator 105 (e.g., via a GPIO pin on the GPU), which receives power from a PCI or similar bus 106 and / or a power supply unit (PSU) 108 and provides power to the GPU 102 via a power rail 127. The signal conditioner and multiplexer 112 receives the GPU power rail 127 as well as the board input power rail 111. The output control signals of the frequency / voltage control system 122 may also generate internal or external control signals that control one or more clock rates of a clock signal generated by a clock generator 107 to synchronously trigger ("clock") various GPU circuits.

[0083] As an example, if the activity level in processor 116 is low, frequency / voltage control system 122 may operate to reduce one or more power supply voltages in order to reduce the amount of power provided to the GPU. If the activity level is high, frequency / voltage control system 122 may increase one or more power supply voltages in order to provide more power to the GPU to meet the demands of its increased processing workload.

[0084] Managing Power

[0085] Figure 3 is a flow chart of an example power control technique using current and predicted GPU power according to certain example embodiments. In some example embodiments, process 300 (i.e., operations 302-312) is performed in hardware circuit components on GPU 102. In some other embodiments, process 300 may include operations performed by other hardware components external to GPU 102 in addition to operations performed on the hardware of GPU 102. In yet another embodiment, process 300 may include operations performed in software running on GPU 102 and / or other parts of system 100 in addition to operations performed by hardware. However, in general, in order to provide sufficient speed performance, Figure 3 The operations shown are performed in hardware, with the exception of deep learning 312 which may be performed in software with hardware acceleration.

[0086] Process 300 may be continuously or continually active during operation of a computer system using a system (eg, system 100 ) and / or a GPU (eg, GPU 102 ).

[0087] Operation 302 monitors the current power used by the GPU. In an embodiment, the monitoring of input power uses ADC 118 to sample the power provided to GPU 102. The monitoring may include ADC 118 sampling the input current and / or voltage to the voltage regulator and converting the sampled result to a digital output of any desired resolution.

[0088] At operation 304, each parallel processor 116 may process the received instructions to provide a predicted power signal 142 for the pending (to-be-executed) instructions. The pending instructions from the various parallel processors 116 may be aggregated and used to predict the GPU power for an upcoming processing cycle during which the pending instructions will be executed (see block 304).

[0089] At operation 306, the difference (error) between the predicted GPU power and the current power is determined. Before determining the error, the predicted power may be scaled to be at the same level as the determined current power.

[0090] At operation 308, GPU 102 determines frequency and / or voltage settings to reduce the error. The determination may include using an adaptive proportional-integral-derivative (3-term) controller ("PID controller") 440 (see below) to determine the settings.

[0091] At operation 310, based on the determination in operation 308, GPU 102 may change (i.e., program a new) clock frequency and / or power voltage to increase or decrease the power drawn by the GPU. For example, GPU 102 may adjust the frequency (or rate) of one or more clocks if it determines that a clock frequency adjustment alone would affect the desired power change. Voltage may be adjusted by offsetting the voltage sent to the voltage regulator to bring the current power back below the limit or to increase it.

[0092] In some embodiments, the parameters of the operation 306 performed using the adaptive PID controller are updated based on a machine learning method (block 312) (which may be implemented in hardware as is known to those skilled in the art). Since the workload in the parallel processor 116 may be diverse and change over short periods of time, a reinforcement learning method utilizing player-judge learning may be used to update the parameters of the operation 306 in conjunction with the adaptive PID controller. Figure 3 The PID parameters may be updated at different time periods during the process. For example, based on the voltage and / or frequency parameters maintained by the hardware, blocks 302-310 may be executed quickly between processing cycles of the processor 116. Figure 3 process, and the deep learning block 312 may execute over many such individual processing cycles and produce an output that is used to more irregularly update the voltage and / or frequency parameters maintained by the hardware.

[0093] In this way, process 300 can proactively control the power input of GPU 102 based on the workload in the parallel processor to optimize processing performance and power consumption. Learning 312 is added to provide a second level of dynamic control to update parameters used to determine GPU 102 power-related settings.

[0094] Enhanced Machine Learning

[0095] Figure 4 and Figure 5 Circuit 400 is schematically illustrated to implement active power management techniques and learning mechanism 150 according to certain example embodiments. Circuit 400 includes ADC 118, filter and scaler circuit 420, comparator or adder 422, state vector register 430, PID controller 440, frequency / voltage controller 450, and learning system 460.

[0096] Circuit 400 may be fabricated on the same substrate that includes a processing system (e.g., GPU 102). Including circuit 400 on the substrate of GPU 102 provides immediate, rapid feedback and response. Unlike reactive systems that respond to changes in current power measurements, the system in the example embodiment allows for proactive adjustments to be made by the processing system because the power control system can anticipate slowdowns or increases in processing between the time instructions are proposed for execution and the time they are actually executed.

[0097] ADC 118 may be configured to monitor power supply voltage and current levels used by a processing system (e.g., GPU 102) including multiple processors 116 (e.g., a programmable streaming multiprocessor). ADC 118 is configured to receive an input signal representing a voltage and / or current provided to the processing system and to generate a digital output signal representing the current power used by the processing system.

[0098] The filter and scaler circuit 420 may receive a prediction signal from each processor 116 indicating how much power instructions are about to be executed or may be required in the near future. For example, a processor 116 may report whether a pending instruction will result in high energy consumption or low energy consumption in a clock cycle. The signal from each processor 116 may be a binary high power / low power flag signal, or it may provide a range of different values ​​to any desired resolution indicating how much power may be required (or may be, if speculative execution is enabled) to execute an instruction in the near future that has not yet been executed but is about to be executed in the instruction pipeline.

[0099] like Figure 5As shown, the processors 116 may each include a pre-decoder 117, which is configured to send (signal) a signal of the possible energy of the instruction to be executed in the future. The filter circuit 420A may be configured to receive an aggregate power signal from all processors 116. The aggregate signal may be generated by an aggregator 422, which in one embodiment includes a combination logic that votes on the aggregate signal or otherwise quickly calculates the aggregate signal to submit to the filter 420A. The signal represents how much power the pending instruction may require when it is executed, and the pending instruction will be executed by the processor 116 library for a predetermined number of clock cycles later. In an example non-limiting embodiment, these signals do not predict how much power will actually be required, but only predict that the instruction may consume more or less power based on its complexity or other instruction characteristics. For example, in an example embodiment, the processor 116 that parses the memory load instruction may send a signal that the upcoming instruction may require low (lower) power. The same processor 116 that parses the calculation instruction may send a signal that the upcoming instruction may require high (higher) power.

[0100] In many embodiments, the processor 116 itself may not "know" how much power is actually consumed executing any given instruction. This is like a long-distance runner who knows roughly how many calories of energy she may consume running a half marathon, but generally cannot predict the energy consumption to calories. There are too many variables (e.g., environmental conditions such as temperature, actual speed rather than planned speed, exact body mass, etc.) that will affect actual energy usage. In addition, the exact power requirements of any given processor 116 will depend on the characteristics of that particular processor, as manufacturing and process variations may result in subtle architectural differences from GPU to GPU or even from one processor to another on the same GPU. The exact power usage also typically depends on the specific software instruction sequence executed by the processor 116.

[0101] In the example shown, the filter and scaler circuit 420 includes one or more filters 420a (e.g., infinite impulse response or IIR filters) to reduce noise in the aggregated signal received from the processor 116. The impulse response of such a filter is "infinite" because there is feedback in the filter; if an impulse (a "1" sample followed by many "0" samples) is input, theoretically an infinite number of non-zero values ​​will be generated. However, other implementations may use FIR filters, combinations of FIR filters and IIR filters, or any other suitable filter design.

[0102] The output of filter 420a is a near future power estimate. This near future power estimate is provided to scaler 420b, which scales the output so that it can be combined with the output of ADC 118 (the scaler may be controlled in part by the magnitude of the ADC output).

[0103] Circuit 400 may further include circuitry (eg, summing node 422) to determine the difference or error between the current power measurement by ADC 118 and the power estimated by filter / scaler 420 to be used for a predetermined number of cycles.

[0104] The state vector register 430 receives / stores the error value to provide a value X(t) that the PID controller 440 can use to determine the frequency and / or voltage settings. In some exemplary embodiments, X(t) takes into account the error values ​​of multiple clock cycles (e.g., the error of the current cycle, one or more past cycles, and / or one or more future cycles). The output X(t) of the state vector circuit 430 may include multiple components. In some embodiments, the output X(t) may provide the error as a function (e.g., a multiplier function, a scaling function, or a unity function). In some embodiments, the state vector circuit 430 may take the error provided in a continuous form and provide a discrete value to the PID controller 440. In an example embodiment, the output X(t) of the state vector circuit 430 may be represented by e(t), Δe(t), Δ2e(t)... The entire vector X(t) may be provided to the learning system 460, and a subset of the state vector (e.g., component e(t)) may be provided to the PID controller 440.

[0105] The PID controller 440 receives a representation of the error between the current power and the predicted power and calculates the output W(t) based on specific parameters stored in the "tuning constants" register 440A by the PID controller 440. As is well known, the PID controller 440 combines proportional control with additional integral and derivative adjustments, which help the controller automatically compensate for changes in the system. As is well known, the PID controller 440 forces feedback to match the set point. Proportional adjustment includes correcting a target that is proportional to the difference; it approaches but never reaches zero error. Integral adjustment accumulates the error generated by the "P" action to increase the correction factor. Since this sometimes causes overshoot, the derivative adjustment is used to minimize the overshoot by slowing down the applied correction factor when approaching the target. In an example embodiment, the PID stage of the PID controller 440 is parameterized using gain values ​​or tuning constants for the proportional, integral, and derivative terms. In an example embodiment, these parameters are stored in the read / write register 440A and can be dynamically changed by the learning system 150 as described below.

[0106] The PID controller 440 output W(t) specifies a change in operating voltage and / or frequency to reduce the error between the current power and the predicted power. When the error is zero, the output of the PID controller 440 stops adjusting the frequency and voltage. The PID controller 440 output W(t) can be selected between (a) frequency adjustment only, (b) voltage adjustment only, and (c) voltage and frequency changes, depending on the time period over which the power should be managed.

[0107] The output W(t) of PID controller 440 is provided to frequency / voltage controller 122 for adjusting the operating frequency and / or voltage of the processing system (eg, GPU 102). The operation of frequency / voltage controller 450 may be performed in hardware and / or software.

[0108] Based on the output W(t), the frequency / voltage controller 122 adjusts the operating frequency and / or voltage to allow the power drawn by the GPU 102 to increase or decrease. For example, if it is determined that a frequency adjustment alone can affect the required power change to reduce the error, the frequency / voltage controller 450 can trigger a frequency change to adjust the frequency (or rate) of one or more clocks 127. The PID controller 440 can determine whether to change only the operating frequency, only the voltage, or both the voltage and the frequency. In some embodiments, the degree of error can determine which operating settings should be changed. For example, if the error is above a predetermined threshold, the PID controller 440 can require a change in the operating voltage. When a high power event is predicted, the operating voltage and frequency can be changed to limit the peak power, thereby improving system-level operation and reliability when operating at maximum efficiency.

[0109] In some embodiments, the frequency / voltage controller 122 may adjust the voltage by offsetting the set voltage sent to the voltage regulator to bring the current power back below the limit or increase.

[0110] As described above, the PID controller 440 allows for fixed parameters or self-tuning parameters of the controller. Since the workload can be diverse and change over short periods of time, reinforcement learning (e.g., player-judge or Q-learning 150) can be used to update the PID controller 440 parameters stored in register 440a. The learning system 150 can also take note of changes in the processing system (e.g., silicon changes) that would otherwise not be accounted for by the pre-set PID controller 440 parameters, preventing the PID controller 440 from overcompensating for the expected power demand.

[0111] The learning system 150 may continuously receive the error X(t) from the state vector 430 and the power measured by the current from the ADC 118. In some embodiments, the learning system 150 may also receive the predicted power from the processor 116 and / or the aggregator 422. In an example embodiment, the learning system 150 adjusts the parameters 440a of the PID controller 440. The learning system 150 outputs K(t), which constitutes the PID control vector, which is updated based on the correction effect to give the dynamic characteristics.

[0112] The learning system 460 includes one or more deep neural networks that periodically update controller parameters based on the effectiveness of correcting errors over time. For example, if the PID 440 is initialized at manufacturing time using tuning constants 440a that are empirically derived based on simulations, they may not exactly match the actual hardware characteristics of any particular GPU 102. However, the learning system 150 can implement reinforcement learning to learn the actual correlation between the state vector 430 values ​​X(t) for predicting the current draw for a particular set of processor instructions and the actual current / power draw of the GPU 102 when the GPU executes those instructions. The learning system 150 can use statistical and probabilistic analysis to determine the more optimal mapping between X(t) and W(t) that the PID controller 440 performs for a variety of different X(t), despite the potential complexity of the output signal W(t), and without being programmed with any preconceived model or mapping of the system.

[0113] In an example embodiment, machine learning 150 may be accelerated by a hardware deep learning accelerator ("DLA") 141, and may perform deep reinforcement learning algorithms, such as player-judge reinforcement learning or Q-learning. See, for example, Konda et al., "Player-judge-like learning algorithms for Markov decision processes," SIAM Journal on Control and Optimization, 38(1): 94-123 (1999); Sutton et al., "Reinforcement Learning: An Introduction" (MIT Press, Cambridge, MA, 1995); and Grondman et al., "A Survey of Player-Judge Reinforcement Learning: Standard and Natural Policy Gradients," IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews (2012). As summarized by Grondman et al.:

[0114] Reinforcement learning is a framework in which an agent (or controller) optimizes its behavior by interacting with an environment. After taking an action in a state, the agent will receive a scalar reward from the environment, which will indicate to the agent the quality of the action. The function that indicates what action to take in a state is called a policy. The main goal of the agent is to find a policy that maximizes the total cumulative reward, also known as the return. By following a given policy and processing rewards, the agent can build an estimate of the return. The function that represents this estimated return is called the value function. Using this value function allows the agent to indirectly use past experience to decide on future actions to take in or around a state… The player-judge approach combines the advantages of the player-only and judge-only approaches. While the parameterized player brings the advantage of computing continuous actions without the need to optimize the value function, the advantage of the judge is that it provides the player with low-variance knowledge of the performance. More specifically, the judge's estimate of the expected return allows the player to update using gradients with lower variance, thus speeding up the learning process. When the judge's estimate is far from accurate, the lower variance is exchanged for a larger deviation at the beginning of learning. The player-judge method generally has good convergence properties compared to the judge-only method.

[0115] Deep neural networks (DNNs) such as learning system 150 developed on processors such as processor 116 have been used in a wide variety of use cases, from self-driving cars to faster drug development, from automatic image annotation in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that simulates the neural learning process of the human brain, constantly learning, constantly getting smarter, and providing more accurate results faster over time. Children are initially taught by adults to correctly identify and classify various shapes, and eventually are able to recognize shapes without any guidance. Similarly, a deep learning or neural learning system needs to be trained in object recognition and classification in order to make it smarter and more efficient in recognizing basic objects, occluded objects, etc., while also assigning context to objects. In general, reinforcement learning is between supervised and unsupervised learning. While in supervised learning, each training example has a target label, in unsupervised learning, there are no labels at all, in reinforcement learning, there are sparse and delayed labels - rewards. Based solely on these rewards, the agent must learn to behave in the environment.

[0116] See: https: / / neuro.cs.ut.ee / demystifying-deep-reinforcement-learning /

[0117] At the simplest level, neurons in the human brain observe the various inputs they receive, assign importance levels to each of these inputs, and pass outputs to other neurons to perform actions. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of the object that the perceptron is being trained to recognize and classify, and each of these features is assigned a certain weight based on the importance of that feature in defining the shape of the object.

[0118] Deep neural network (DNN) models consist of many layers of connected perceptrons (like nodes) that can be trained with large amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DLL model breaks down an input image of a car into its different parts and looks for basic patterns such as lines and angles. The second layer assembles the lines to look for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image that identify the model as a specific car brand.

[0119] Once a DNN is trained, it can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on a check deposited into an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, recognizing and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0120] During training, data flows through the DNN in a forward propagation phase until a prediction is produced indicating a label corresponding to the input. If the neural network does not correctly label an input, the error between the correct label and the predicted label is analyzed and the weights of each feature are adjusted in a backward propagation phase until the DNN correctly labels that input and other inputs in the training data set. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by processors 116. Inference, which is less computationally intensive than training, is a latency-sensitive process in which a trained neural network is applied to new inputs it has never seen before to classify images, translate speech, and generally infer new information.

[0121] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require large amounts of floating point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, processor 116 provides a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications such as those described above.

[0122] Example Unrestricted Reinforcement Learning Proportional-Integral-Derivative Controller

[0123] Figure 6 An example more detailed architecture of a reinforcement learning based PID controller 440 is shown. In this example implementation, the input signal y_ref(t) represents the reference power we want to manage, and y(t) is an output signal representing the sum of the current sensor power + the expected future power increase that will be observed due to high energy instructions. Adder 422' calculates the difference between y_ref(t) and y(t) to generate an error signal e(t) to be applied to the state vector 430. Therefore, the state vector x(t) is a vector derived from e(t), where: x(t) = [x1(t), x2(t), x3(t)..xn(t)] = [e(t), Δ(e(t)), Δ^2(e(t)),..]: e(t) = y_ref(t)–y(t), Δ(e(t)) = e(t)–e(t-1), Δ^2(e(t)) = e(t)–2*e(t-1)+e(t-2), and so on.

[0124] The PID controller 440 will use the elements of this state vector, typically for example, e(t) will go to the block incremental PID controller 440. The output of the PID controller 440 is Δu(t), which is applied to the input of a process 472 representing the frequency-voltage controller 122 and the associated power regulation mechanism. The Z-1 (memory) operator 470 provides hysteresis so that the PID controller 440 output Δu(t) is used to adjust the current u(t) signal applied to the control process 472. The Z-1 operator 470 stores u(t) and provides it as u(t-1) to the input of an adder 474, which adds Δu(t) to u(t-1) to produce u(t).

[0125] The learning block 150 includes players 476, judges 480, and a self-adaptive module ("SAM") 478. Typically, the system 150 is a neural network based model that will generate update coefficients K(t) for the PID controller 440. The input to the judges 480 is r(t), also known as the reward for an entity in a reinforcement learning system. In this particular case, r(t) is a scalar value derived from e(t) and e(t-1). The goal of learning is to maximize r(t).

[0126] This particular arrangement causes the player 476 to provide its output K'(t) to a self-adaptive module ("SAM") 478, which generates K(t) and provides this signal as a coefficient input to the PID controller 440. In this example, these coefficients constitute a vector of PID parameters.

[0127] Meanwhile, judges 480 generate output V(t) based on inputs x(t) and r(t), and apply output V(t) to SAM 478, which adapts the judges' output and transforms K'(t) into K(t).

[0128] Dual control loop architecture

[0129] Thus, PID controller 440 and learning system 150 provide two control loops, a first including PID controller 440 and a second including the learning system running on top of the first loop (at a different time period than the first loop). These two control loops provide self-learning and self-adjustment of the parameters used by PID controller 440 to adjust the power control parameters of GPU 102 based on future power usage estimates provided by processor 116 (as a result of the processor parsing the instruction format it is about to execute).

[0130] In an example non-limiting embodiment, the PID controller 440 loop operates continuously and is independent of the learning system 150 loop, but may be updated periodically as determined by the learning system loop. The learning system loop observes the operation of the PID controller 440 loop to see if there is a difference (e.g., error reduction) in the currently set PID controller 440 weighting coefficients. If there is no difference, or the error reduction can be improved, the learning system loop 150 may modify one or more coefficients used by the PID controller 440. The learning system 150 loop may update the PID controller coefficients when the PID controller 440 is not updating the frequency and / or voltage settings.

[0131] As described above, in some exemplary embodiments, the learning system 150 may receive additional information from other components of the processing system. For example, the learning system 460 may receive information about pending instructions scheduled for execution from the processor 116. The learning system 460 may use this additional information to help it determine and update the coefficients of the PID controller 440.

[0132] Example of a non-restrictive and scalable reinforcement learning proportional-integral-derivative system

[0133] In an example implementation, the reinforcement learning PID system can be extended to include one or more additional PID controllers and / or one or more additional learning blocks to control the system based on separate metrics. In the context of predictive power management, the scalable reinforcement learning PID system can be used to manage performance (separate metric) and power (separate metric) to achieve the highest potential operating point.

[0134] Figure 7 is a schematic diagram of a scalable reinforcement learning proportional-integral-derivative system. Figure 7In the embodiment of the present invention, one or more PID controllers 440 are used to control the process 472 , and one or more learning systems 150 update coefficients of the one or more PID controllers 440 .

[0135] like Figure 7 As shown, the process 472 may be controlled based on the summed output of the PID controllers 440. Each PID controller may manage a separate measure of setting one or more common parameters of the process 472.

[0136] exist Figure 7 In the example shown, each PID controller 440 is coupled to a respective learning system 150, which determines coefficients for the coupled PID controller. Learning system 155 determines the coefficients for PID controller 440 based on one or more outputs of process 472. In other examples, learning system 150 determines the coefficients based on one or more error signals input to PID controller 440. In some examples, a single learning system (e.g., RL agent 1) can determine coefficients for two or more PID controllers (e.g., PID controller 1 and PID controller 2).

[0137] Figure 8 An example architecture of a scalable reinforcement learning based PID system including two PID controllers and two learning systems is shown. In this example implementation, a first PID controller 440A and a second PID controller 440B are used to control a process 472, which can represent a frequency-voltage controller and an associated power regulation mechanism. The input signal θ ref and x ref We can represent the individual metrics we want to manage, θ(t k ) and x(t k ) can represent the current metric value and the estimated future increase in the metric value. For example, x(t k ) can represent the sum of the current sensor power + the estimated future power increase that will be observed due to high energy instructions, θ(t k ) may represent the sum of current performance + estimated future performance increase that will be observed due to high energy instructions. In some example implementations, the first metric may represent the voltage of process 472 and the second metric may represent the frequency of process 472.

[0138] The first PID controller 440A receives the first metric error signal e p (t k ), which represents θ ref and θ(t k The second PID controller 440B receives the second error signal e c (t k ), which means xref and x(t k The first PID controller 440A and the second PID controller 440B respectively output values ​​u p (t k ) and u c (t k ), which are determined based on the received error and the coefficient K(t) set in the controller. Figure 8 As shown, the output value u p (t k ) and u c (t k ) are added and provided to process 472.

[0139] When a single metric sets one or more common parameters of process 472, the output value u p (t k ) and u c (t k As an example, if the operating clock is set based on performance and power (because the clock affects both), the output will be used to drive / set common parameters of the control process.

[0140] In other examples, the output value u can be p (t k ) and u c (t k ) is directly provided to process 472 for determining the output value u p (t k ) and u c (t k )How to drive / set parameters of control process 472.

[0141] like Figure 8 As shown, the first PID controller 440A and the second PID controller 440B can be coupled to respective learning systems to update the coefficients of the controllers. The first learning system 150A generates one or more update coefficients K for the first PID controller 440A. PP ,K IP and / or K DP The second learning module 150B generates one or more update coefficients K for the second PID controller 440B PC ,K IC and / or K DC The first learning block 150A and the second learning block 150B receive a state vector (n) representing a discrete value output by the process 472. 1 (t k ), n 2 (t k ), n 3 (tk ), n 4 (t k )) is one or more components. The state vector value n 1 (t k ), n 2 (t k ), n 3 (t k ) and n 4 (t k ) Output θ(t k ), x(t k )and Export in.

[0142] exist Figure 8 In the example shown, the first learning block 150A receives the state vector value n 1 (t k ) and n 2 (t k ), the second learning block 150B receives the state vector value n 1 (t k ), n 2 (t k ), n 3 (t k ) and n 4 (t k ).although Figure 8 The learning blocks are shown receiving different sets of values, but the first learning block 150A and the second learning block 150B may receive the same set of values ​​or different sets of values.

[0143] In other examples, learning systems 150A and / or 150B may use the error signal e p (t k ) and / or error signal e c (t k ) to update the coefficients of the first and / or second PID controller.

[0144] Example of unrestricted reinforcement learning feedback loop control method

[0145] Fig. 9 is a flow chart of a reinforcement learning feedback control method according to certain example embodiments.

[0146] For the launcher, operation 510 initializes one or more parameters. For example, these parameters may include parameters used by PID controller 440 to affect the power control parameters of GPU 102.

[0147] Operation 520 senses the error, for example, by generating an error signal e(t) for application to the state vector 430, such as provided by calculating the difference between y_ref(t) and y(t).

[0148] Operation 530 uses reinforcement learning to detect whether the sensed error is a system error that needs to be corrected. Reinforcement learning is able to link immediate actions to the delayed rewards they produce. Like humans, reinforcement learning algorithms sometimes need to wait for a while to see the results of their decisions. Therefore, they can operate in a delayed return environment where it is difficult to understand which action will lead to which result over many time steps. Therefore, reinforcement learning algorithms can perform better in ambiguous environments, such as power control with unknown factors (e.g., the parallelism of specific instructions / threads / thread warps of a specific software application or its functions, the number of GPU transistors that will be activated and deactivated based on such parallel operations, etc.), while choosing between any number of possible actions. In an example operation of a non-limiting embodiment, the judge neural network of the reinforcement machine learning system has been trained for previous power adjustment operations in response to previous sensed errors and has evaluated the accumulated rewards, that is, the value assessment of how favorable it is in a specific state. If the neural network determines that the evaluated reward is high based on the sensed error, the state vector is unchanged and the "sensing error" operation 520 is repeated continuously. However, if the neural network detects that the reward is low based on the sensed error, additional processing is invoked to change the state vector (described below).

[0149] In order to change the state vector when the evaluated reward is low, the example non-limiting embodiment performs operation 540, i.e., a neural network-based state vector-based parameter calculation (in other embodiments, the state vector-based calculation can be performed in other ways, such as increasing or decreasing a certain amount, etc.). The calculation may include, for example, calculating the coefficient K'(t) of the PID controller 440. In an example non-limiting embodiment, the state vector-based parameter calculation can be performed by the player 476 based on the state vector x(t) derived from the error signal e(t).

[0150] In one example non-limiting embodiment, operation 550 performs random perturbations to convert the calculated coefficients K'(t) to K(t). Based on the output V(t) from the judges 480, the adaptive module ("SAM") 478 can perform the conversion of the coefficients K'(t) to K(t). In operation 560, the new coefficients K(t) derived by the neural network are applied to the PID controller 440.

[0151] Applying the new coefficients K(t) to the PID controller 440 results in a shift in the operating point of the GPU power setting (block 570). The shift in operating point may include adjusting the output frequency of the frequency-voltage controller 122 and / or adjusting the output voltage using a power adjustment mechanism. The result of the process performed by the frequency-voltage controller 122 and the associated power adjustment mechanism sets the new frequency and / or voltage operating point of the GPU. The GPU then processes the instruction using the new operating point (block 580). At the same time, the neural network learns from the effects of this operation and Fig. 9 This additional learning is used in the next iteration of the process.

[0152] Overcurrent protection

[0153] In some exemplary embodiments, the frequency / voltage controller 450 may delay, preempt or prevent the execution of pending instructions in one or more processors (e.g., SM). For example, the frequency / voltage controller 450 and / or the machine learning 150 may receive an output from the PID controller 440 and determine when the execution of the instruction will cause the frequency and / or voltage setting to exceed a predetermined limit (e.g., a safety limit). Based on this determination, the frequency / voltage controller 450 may send a signal to delay or stop processing the pending instruction in one or more cycles. The signal to delay or stop processing the instruction may be sent to one or more processors via a conventional preemption mechanism known to those skilled in the art. The frequency / voltage controller 450 may determine the number of processors 116 that need to delay processing of high-energy instructions to avoid exceeding a predetermined safety limit. For example, when it is determined that the execution of the pending instruction in the processor 116 will cause the power setting to exceed a predetermined limit, which may cause overheating, power failure or other adverse effects, the frequency / voltage controller 450 may send a delay signal to one or more processors so that all high-energy consumption instructions are not executed in a given clock cycle. Such load balancing can make high-power calculations uniform over time while minimizing the impact on overall speed performance.

[0154] The exemplary embodiments of the present disclosure are not limited to the use of Figure 4 and Figure 5 PID controller and player-judge learning or Q-learning as shown. Other feedback controllers and learning systems may be used with the exemplary embodiments disclosed in this application. For example, a PID or other neural network controller (if fast enough) may be used to determine changes in operating frequency and / or voltage that should be applied to a GPU based on the error between current and predicted power.

[0155] As described above, conventional GPUs are limited to using reactive mechanisms to manage the power of the GPU. However, such reactive mechanisms make the internal components of the GPU vulnerable to electrical reliability issues (e.g., internal components become damaged and / or the GPU shuts down and falls off the communication bus). Embodiments of the present disclosure provide active sensing of instructions to predict possible load step / release events, which allows knowledge of an impending load step / release event several clock cycles later, before the event actually occurs. Predicting a load step / release event allows the power settings of the GPU to be adjusted (e.g., by changing voltage and / or frequency) before the event occurs to prevent peak power conditions. Moving the voltage and / or frequency in this manner can reduce electrical reliability issues in a shorter time than other methods, but without compromising performance. In addition, a learning system running in parallel with the PID controller 440 provides dynamic adjustment of parameters for determining that the operating voltage and / or frequency should be changed to predict and prevent such undesirable peak power conditions.

[0156] According to an embodiment, a GPU including an active power management system provides higher reliability than other methods over a shorter period of time without compromising performance, and can therefore be used in computing environments that require high reliability. According to an embodiment, a GPU and a SoC including one or more GPUs can be implemented in data centers, server environments, gaming computers, laptops, and other computing environments. The embodiments can also be used in environments such as autonomous vehicles, robots, and the like. For example, an embodiment can improve the operational reliability of autonomous vehicle applications by ensuring that a GPU does not temporarily fall off a communication bus due to an overcurrent condition. The embodiments can also improve the power consumption of autonomous (or other) vehicles by achieving faster power management response.

[0157] However, embodiments are not limited to GPUs and SoCs including GPUs.Other embodiments may provide an on-chip ADC or other type of processor on a CPU chip.

[0158] Embodiments of the subject matter and functional operations described herein may be implemented in one or more of the following embodiments: digital electronic circuitry; tangibly embodied computer software or firmware; computer hardware, including the structures disclosed in this specification and their structural equivalents; and combinations thereof. The embodiments may be implemented as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device (i.e., one or more computer programs) or to control its operation. The computer storage medium may be one or more of the following: a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, and combinations thereof.

[0159] Each of the items cited above is incorporated by reference for all purposes as if expressly stated.

[0160] While the invention has been described in what are presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A GPU, comprising: parallel processors, each parallel processor including an instruction decoder that decodes instructions prior to execution and signals expected power usage of instruction execution; a power controller that adjusts power provided to the parallel processors responsive at least in part to the signaled expected power usage; and a neural network that adapts the adjustments performed by the power controller, Wherein, the neural network performs reinforcement learning to mitigate peak power scenarios.

2. The GPU of claim 1 , wherein the instruction decoder comprises an instruction look-ahead pre-decoder that pre-decodes instructions to be executed in the future and estimates how much power will be used to execute the instructions, and each of the parallel processors comprises an additional instruction decoder that decodes the instructions for execution.

3. The GPU of claim 1, wherein the power controller is configured to reduce power to avoid peak power overshoot.

4. The GPU of claim 1 , wherein the neural network controls the parallel processors to modify their instruction execution to reduce peak power requirements.

5. A processing system comprising: Multiple processors; as well as Power management circuit, configured as: comparing the power currently being used by the plurality of processors with a predicted power estimated to be used by the plurality of processors in the future; generating one or more control signals in response to the comparing to vary operating powers of the plurality of processors; as well as Reinforcement learning is performed to adapt the generating of the one or more control signals.

6. The processing system of claim 5, wherein the reinforcement learning uses player-judge-based machine learning or Q-learning machine learning.

7. The processing system of claim 5, wherein the power management circuit comprises a proportional-integral-derivative (PID) controller configured to receive the difference result of the comparison and generate the one or more control signals to change the operating frequency and / or voltage of the processing system.

8. The processing system of claim 7, wherein the reinforcement learning modifies operating coefficients of the PID controller.

9. The processing system of claim 7, wherein the PID controller generates the one or more control signals at each clock cycle, and the reinforcement learning modifies the operation coefficients of the PID controller at intervals exceeding the clock cycle.

10. The processing system of claim 5, wherein: Each processor is configured to pre-decode instructions received by the corresponding processor and generate a signal based on the pre-decoded instructions, the signal estimating an energy consumption level of the instructions to be processed; and The power management circuit is further configured to aggregate signals indicative of energy consumption levels from each processor and predict power based on the aggregated signals.

11. The processing system of claim 5, wherein the power management circuit is configured to adjust an operating frequency and an operating voltage of the processing system based on the generated one or more control signals.

12. The processing system of claim 5, wherein the power management circuit is configured to control one or more processors of the plurality of processors to delay execution of one or more pending instructions based on the generated one or more control signals.

13. The processing system of claim 5 , wherein the power management circuit comprises an analog-to-digital converter (ADC) configured to receive an input signal representing a voltage and / or current being provided to the processing system and to generate a digital output signal representing the power being used by the processing system based on the input signal.

14. The processing system of claim 5, wherein the plurality of processors and the power management circuit are disposed on a same substrate.

15. A method of dynamically controlling voltage and frequency settings of a plurality of processors configured to receive and execute instructions, the method comprising: measuring power currently used by the plurality of processors; predicting power of the plurality of processors by examining instructions to be executed by the plurality of processors; determining the error between the current power and the predicted power; generating one or more signals to change the operating frequency and / or the operating voltage to reduce the error; as well as Reinforcement learning is performed to adapt the generation of the one or more signals.

16. The method of claim 15, wherein generating the one or more signals comprises: A proportional-integral-derivative (PID) controller is used to generate the one or more signals based on the error to change an operating frequency and / or an operating voltage provided to one or more of the plurality of processors.

17. The method of claim 16, wherein the PID controller generates the one or more signals at intervals corresponding to clock cycles, and the reinforcement learning modifies coefficients of the PID controller at intervals exceeding the intervals corresponding to the clock cycles.

18. The method of claim 16, wherein the reinforcement learning modifies coefficients of the PID controller.

19. The method according to claim 15, further comprising: generating, for each processor, a signal representing an energy consumption level of pending instructions to be executed in clock cycles; as well as The predicted power is determined by aggregating the signals representing energy consumption levels of instructions to be processed in corresponding clock cycles.

20. A controller comprising: a power monitor configured to monitor voltage and / or current being supplied to the plurality of processors; circuitry configured to receive signals from the plurality of processors representing estimated energy consumption for future execution of instructions and to predict power that the plurality of processors will use to execute the instructions in the future based on an aggregation of the received signals; as well as A self-learning and self-adjusting controller is configured to control power provided to the plurality of processors in response to the monitored voltage and / or current and the predicted power.

21. The controller of claim 20, wherein the self-learning and self-adjusting controller comprises: a proportional-integral-derivative (PID) controller configured to determine power to be provided to the plurality of processors based on the coefficients, and At least one neural network is configured to perform reinforcement machine learning to modify the coefficients.

22. The controller of claim 20, wherein the self-learning and self-tuning controller is disposed on the same substrate as the processor.

23. The controller of claim 20, wherein the self-learning and self-tuning controller is further configured to send a signal to one or more of the plurality of processors to delay execution of at least one instruction based on the determined power to be provided to the plurality of processors.

24. The controller of claim 20, wherein the self-learning and self-adjusting controller comprises: a proportional-integral-derivative (PID) controller configured to determine power to be provided to the plurality of processors based on the coefficients, and At least one neural network is configured to perform player-judge reinforcement machine learning to modify the coefficients based on a difference between energy provided to the plurality of processors and an estimated energy consumption for future execution of instructions.

25. The controller of claim 20, wherein the self-learning and self-adjusting controller comprises: a plurality of proportional-integral-derivative (PID) controllers, each PID controller generating a control signal based on a different metric affecting operation of the plurality of processors; as well as a plurality of neural networks, each of which is configured to perform reinforcement machine learning to modify the coefficients of a corresponding PID controller, Wherein the power provided to the plurality of processors is controlled based on a sum of the outputs of the PID controllers.

Images