Reducing voltage droop by limiting assignment of work blocks to computing circuits

Abstract

An apparatus and method for efficiently managing a voltage drop among a replica computing circuit of an integrated circuit. In various implementations, an integrated circuit includes a plurality of copy computing circuits, each copy computing circuit including circuitry for processing tasks grouped into work blocks. When the scheduling window has started, the scheduler determines a value for a threshold number of idle computing circuits that can be simultaneously activated based on one or more of: a number of active computing circuits, an operating clock frequency, a measured operating temperature, a number of work blocks to be processed, and an application identifier. If the scheduler determines that there is a count of idle computing circuits equal to or greater than the threshold number of idle computing circuits, the scheduler limits the number of idle computing circuits that can be activated at a time to the threshold number.

Description

Background Technology

[0001] Description of related technologies

[0002] Various computing devices utilize heterogeneous integration, which integrates multiple types of integrated circuits (ICs) to provide system functionality. These functions include: audio / video (A / V) data processing; other high data-parallel applications in medical and commercial fields; processing instructions based on a general-purpose instruction set architecture (ISA); digital, analog, mixed-signal, and radio frequency (RF) functions, and more. Various options exist for system packaging to integrate these multiple types of ICs. In some computing devices, system-on-a-chip (SoC) is used, while in others, smaller and higher-volume chips are packaged as larger chips in multi-chip modules (MCMs). In still other computing devices, three-dimensional integrated circuits (3DICs) utilize die stacking technology and silicon interposers to vertically stack two or more semiconductor dies in a system-in-package (SiP).

[0003] Regardless of the system package choice, voltage drop in modern ICs has become an increasingly serious design problem with each generation of semiconductor chips. Voltage drop constraints are not only a problem for portable computers and mobile communication devices, but also for high-performance superscalar microprocessors, which include multiple processor cores or multiple cores and multiple pipelines within a single core. The geometry of devices and metal paths on each generation of cores is decreasing. Superscalar design increases the density of integrated circuits on a die with multiple pipelines, larger caches, and more complex logic. Therefore, the number of nodes and buses switching per clock cycle increases significantly.

[0004] Parasitic inductance increases transmission line effects on the chip, such as ringing and reduced propagation delay. Additionally, if a power pin serves all row buffers on a wide bus, simultaneous switching of that bus results in a significant voltage drop. This voltage drop ΔV is proportional to the expression L di / dt, where L is the parasitic inductance and di / dt is the time rate of change of current consumption. Significant voltage drops occur if a large number of nodes are switched simultaneously in addition to the bus. Nodes holding logic high values ​​will now experience a voltage drop that reduces their voltage value below a minimum threshold. For memories and latches without recovery circuitry, the stored values ​​will be lost.

[0005] One way to reduce voltage drop is to reduce the operating clock frequency of the integrated circuit. However, performance will degrade. Furthermore, while node capacitor switching can be reduced for certain blocks and / or circuits within a semiconductor chip by disabling the clock to these areas during off-peak hours using a qualified enable signal, simultaneous switching of numerous nodes and signal lines occurs again during the operation of these circuits, and the voltage drop problem persists. Another way to reduce voltage drop is to place an external capacitor between the power leads. This external capacitor creates a passive bypass, which reduces power line oscillations caused by external inductors. However, it does not significantly reduce oscillations caused by internal inductors.

[0006] Another way to reduce inductive effects is to place an on-chip capacitor between the internal power leads. This capacitor acts as a bypass in the same way as the external capacitor. However, for it to function, the internal capacitor is very large, which requires a significant portion of the chip area. This approach is undesirable when minimizing die area is required.

[0007] In view of the above, there is a need for methods and systems for efficiently managing voltage drops in integrated circuits. Attached Figure Description

[0008] Figure 1 It is a general block diagram of a device for managing voltage drops in the replication computing circuit of an integrated circuit.

[0009] Figure 2 It is a general block diagram illustrating the voltage drop curve of the power supply reference level of an integrated circuit.

[0010] Figure 3 It is a general block diagram illustrating the number of active computing circuits of an integrated circuit changing over time.

[0011] Figure 4 This is a general block diagram of a method for efficiently managing voltage drops in multiple computing circuits of an integrated circuit.

[0012] Figure 5 It is a general block diagram of a computing system that efficiently manages the voltage drop of multiple computing circuits in an integrated circuit.

[0013] Figure 6 It is a general block diagram of an integrated circuit that efficiently manages voltage drops across multiple computing circuits.

[0014] Figure 7 It is a general block diagram of a scheduler that manages voltage drops in the replication computing circuit of an integrated circuit.

[0015] Figure 8It is a general block diagram of a system-in-package (SiP) that manages the voltage drop of multiple computing circuits in an integrated circuit.

[0016] While the invention may have various modifications and alternatives, specific embodiments are shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the drawings and the detailed description thereof are not intended to limit the invention to the specific forms disclosed, but rather, the invention covers all modifications, equivalents, and alternatives falling within the scope of the invention as defined by the appended claims. Detailed Implementation

[0017] In the following description, numerous specific details are set forth to provide a thorough understanding of the invention. However, those skilled in the art will recognize that the invention can be practiced without these specific details. In some instances, well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring the invention. Furthermore, it should be understood that, for the sake of simplicity and clarity, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some elements are enlarged relative to other elements.

[0018] Apparatus and methods for efficiently managing voltage drops across multiple computing circuits of an integrated circuit are envisioned. In various specific implementations, the integrated circuit includes multiple replicated computing circuits, each including circuitry for processing work blocks. Tasks performed by the integrated circuit are grouped into work blocks, where a "work block" is a unit of work executed atomically. The granularity of a work block can include a single instruction of a computer program, multiple instructions, or wavefronts comprising multiple work items to be executed concurrently on multiple execution channels of the computing circuit. A scheduler allocates work blocks to idle computing circuits, which are then activated and process the allocated work blocks.

[0019] When the scheduling window begins, the scheduler determines a threshold number of idle computing circuits that can be allowed to be active simultaneously. In some implementations, the scheduler determines this threshold number based on one or more of the following: the number of active computing circuits, the total number of computing circuits, the number of stopped computing circuits, the number of computing circuits that have recently completed executing a work block, the operating clock frequency, the measured operating temperature, the number of work blocks pending processing, any measured pre-existing voltage drop, the queue utilization value of the queue storing the allocated work blocks, and an application identifier specifying the type of application being processed by the corresponding computing circuit (and therefore the type of work block). If the scheduler determines that there is a count of idle computing circuits equal to or greater than the threshold number, the scheduler allocates a certain number of work blocks equal to that threshold number to the idle computing circuits. In other words, the scheduler limits the number of idle computing circuits that can be activated at one time to this threshold number.

[0020] In some implementations, the scheduler modifies the scheduling rate. During a predetermined time period, the scheduler limit can be the total number of idle computing circuits allocated work blocks and activated to process those blocks. In one implementation, the scheduler defines this time period as a specific number of clock cycles. At the end of this time period (e.g., when the time period since the most recent scheduling window expires), the scheduler re-evaluates the scheduling based on updated parameters. Figures 1 to 8 Further details of these techniques for reducing voltage drops in multiple computing circuits of an integrated circuit are provided in the following description.

[0021] refer to Figure 1 This diagram illustrates a general block diagram of a device 100 for managing voltage drops in a replica computing circuit of an integrated circuit. In the illustrated embodiment, device 100 includes a control block 140 and at least two modules, such as modules 110A-110B, which include circuitry configured to (e.g., based on instruction execution, signal detection, data movement, signal and / or data generation, etc.) perform (or “run”) tasks. Tasks are grouped into work blocks. As used herein, a “work block” is a unit of work executed atomically. In some embodiments, the granularity of the work block comprises a single instruction of a computer program. In other embodiments, the granularity of the work block comprises multiple work items to be executed concurrently on the hardware of multiple execution channels of the computing circuitry. In such embodiments, the work block may also be referred to as a “wavefront,” which will be described further later. Control block 140 includes a scheduler 142 and a power manager 144. Control block 140 sends operating parameters and assigned tasks 150 to module 110A. Control block 140 also sends operating parameters and assigned tasks 152 to module 110B. Control block 140 receives performance metrics 130 from module 110A and performance metrics 132 from module 110B.

[0022] Module 110A includes partition 120A, which includes semiconductor dies 122A-122B (or dies 122A-122B). Module 110B includes partition 120B, which includes dies 122C-122D. In some embodiments, each of dies 122A-122B and 122C-122D includes one or more computing circuits. For example, die 122A includes computing circuits 124A and 124B (or computing circuits 124A-124B), and die 122C includes computing circuits 124C and 124D (or computing circuits 124C-124D). Although not shown, dies 122B and 122D may also include one or more computing circuits. In various embodiments, the hardware (such as circuitry) of each of dies 122B and 122C-122D is an instantiated copy of the circuitry of die 122A. Although only two modules 110A-110B are shown, and only two dies are shown in each of partitions 120A-120B, other numbers of modules and computing circuits used by device 100 are possible and conceivable, and these numbers are based on design requirements.

[0023] Power manager 144 assigns a power domain to each of modules 110A-110B. In some embodiments, each of modules 110A-110B uses a specific power domain. In such embodiments, the operating parameters of information 150 and 152 are separate values. In other embodiments, modules 110A-110B share the same power domain. In such embodiments, the operating parameters of information 150 and 152 are the same values. Depending on the embodiment, power manager 144 selects the same power management state or a corresponding power management state for each of modules 110A-110B. As used herein, a "power management state" is one of a plurality of "P states" or a plurality of power performance states, which include a set of operating parameters such as operating clock frequency and operating supply voltage.

[0024] Each power domain includes operating parameters for at least the P-state, such as at least the operating supply voltage and the operating clock frequency. Each power domain also includes control signals for enabling and disabling connections to the clock generation circuitry and the power supply reference. These control signals are also included in information 150 and 152. In one embodiment, the "active computing circuitry" in computing circuits 124A-124B and 124C-124D uses non-zero values ​​of the operating clock frequency and the operating supply voltage to execute the assigned work block. The "idle computing circuitry" in computing circuits 124A-124B and 124C-124D does not have an assigned work block and is typically clock-gated, or otherwise disables the connection to the clock generation circuitry. In some embodiments, the idle computing circuitry disables the connection to the power supply reference level (or power rail). For example, the power switch is deactivated, which disconnects the idle computing circuitry from the power rail. In other embodiments, the idle computing circuitry is clock-gated only, and the idle computing circuitry maintains a connection to its power supply reference level.

[0025] In some embodiments, the functionality of device 100 is included as one of a plurality of dies on a system-on-a-chip (SOC). In other embodiments, the functionality of device 100 is implemented by a plurality of separate dies that have been fabricated on separate silicon wafers and placed in a system package called a multi-chip module (MCM). Other components of device 100 are not shown for illustrative purposes. For example, a memory controller, one or more input / output (I / O) interface circuits, an interrupt controller, one or more phase-locked loops (PLLs) or other clock generation circuits, a level-one or multi-level cache memory subsystem, and various other computing circuits are not shown, but they may be used by device 100. In various embodiments, device 100 is used in desktop computers, portable computers, mobile devices, servers, peripheral devices, etc. Device 100 is capable of communicating with an external general-purpose central processing unit (CPU) that includes circuitry for executing instructions according to a predefined general-purpose instruction set architecture (ISA). Device 100 is also capable of communicating with various other external circuits, such as one or more of digital signal processors (DSPs), display controllers, various application-specific integrated circuits (ASICs), multimedia engines, etc.

[0026] As previously described, tasks performed by device 100 can be grouped into work blocks, where a "work block" is a unit of work executed atomically. The granularity of a work block can include a single instruction of a computer program, and this single instruction can also be divided by device 100 into two or more micro-ops. The granularity of a work block can also include one or more instructions of a subroutine. When computing circuits 124A-124B and 124C-124D are implemented as Single Instruction Multiple Data (SIMD) circuits, the granularity of a work block can also include wavefronts of circuits allocated to multiple execution channels of these computing circuits. In such embodiments, a particular combination of the same instruction and a particular data item among multiple data items is referred to as a "work item." A work item is also referred to as a thread. In one embodiment, each of computing circuits 124A-124B and 124C-124D is a SIMD circuit comprising 64 execution channels. Therefore, each of the computing circuits 124A-124B and 124C-124D (or SIMD circuits) is capable of processing 64 threads simultaneously. In other specific implementations, the computing circuits 124A-124B and 124C-124D include another type of circuitry that provides additional functionality when executed on an allocated work block of another type.

[0027] In one embodiment, scheduler 142 receives work blocks to be allocated to computing circuits 124A-124B and 124C-124D, and does so based on load balancing. In another embodiment, scheduler 142 is the command processor of a graphics processing unit (GPU), the work blocks are wavefronts, and scheduler 142 retrieves these wavefronts from a buffer, such as system memory. Other circuitry, such as a general-purpose central processing unit (CPU), stores wavefronts in a buffer and sends an instruction to device 100 specifying that a wavefront to be processed is stored in that buffer. In other embodiments, scheduler 142 is included in another type of circuitry besides the GPU, and scheduler 142 receives work blocks from another type of circuitry besides the CPU.

[0028] In some embodiments, scheduler 142 allocates work blocks to partitions 120A and 120B in a cyclical manner. Work blocks allocated to computing circuits 124A-124B on die 122A are received by scheduler 125. Work blocks allocated to computing circuits 124C-124D on die 122C are received by scheduler 127. The following discussion describes additional scheduling steps, such as allocating work blocks to computing circuits 124A-124B and 124C-124D. Although the following discussion describes these additional scheduling steps performed by schedulers 125 and 127, in other embodiments, these schedulers 125 and 127 are not present, and subsequent additional scheduling steps are performed by scheduler 142. In still other embodiments, these schedulers 125 and 127 are not present, and subsequent additional scheduling steps are performed by schedulers (not shown) within partitions 120A and 120B.

[0029] In one embodiment, the operating parameters of modules 110A and 110B may differ from each other by a value exceeding a threshold. In one example, modules 110A and 110B share one or more power rails providing one or more power reference levels, but module 110B may operate at half or a quarter of the clock frequency used by module 110A. It may be advantageous for scheduler 142 to perform additional scheduling steps that allocate work blocks to computing circuits 124A-124B and 124C-124D in a manner that reduces voltage drop across the one or more power rails used by modules 110A-110B. However, in such an embodiment, the communication latency between scheduler 142 and computing circuits 124A-124B and 124C-124D is below a threshold. For example, computing circuits 124A-124B and 124C-124D can send queue utilization values ​​for parameters 126 and 128 to scheduler 142 with a latency below the threshold.

[0030] In other embodiments, the operating parameters of modules 110A and 110B differ from each other by less than a threshold value, allowing individual schedulers 125 and 127 to use overlapping scheduling windows. This overlap allows schedulers 125 and 127 to allocate work blocks to corresponding computing circuits in computing circuits 124A-124B and 124C-124D in a manner that minimizes voltage drop on one or more power rails shared by modules 110A-110B. In some embodiments, the circuitry of scheduler 125 allocates work blocks received from scheduler 142 to computing circuits 124A-124B of die 122A for execution. As shown, scheduler 125 sends work blocks as part of information 126 to computing circuits 124A-124B of die 122A. Similarly, the circuitry of scheduler 127 allocates received work blocks to computing circuits 124C-124D of die 122C for execution. As shown in the figure, scheduler 127 sends work blocks as part of information 128 to computing circuits 124C-124D on die 122C. In various specific implementations, schedulers 125 and 127 allocate work blocks to the corresponding computing circuits in computing circuits 124A-124B and 124C-124D in a manner that reduces voltage drop across one or more power rails used by modules 110A-110B.

[0031] Each of schedulers 125 and 127 is capable of determining that a corresponding scheduling window has begun. The following discussion pertains to scheduler 125. However, scheduler 127 includes circuitry that performs similar steps. As previously described, in other embodiments, scheduler 142 includes circuitry that performs the following steps, but concurrently uses the operating parameters of both information 150 and 152, and the queue utilization of both information 126 and 128. To determine that a scheduling window has begun, in one embodiment, scheduler 125 determines that a period of time has elapsed since the most recent allocation of work blocks to the computing circuits 124A-124B of die 122A. In response to determining that a scheduling window has begun, scheduler 125 determines a threshold number of idle computing circuits in computing circuits 124A-124B that can be allowed to be activated simultaneously. Idle computing circuits are activated when connections to one or more of the clock generation circuits and power rails are re-enabled. In some implementations, scheduler 125 determines the threshold number of idle computing circuits based on one or more of the following: the number of active computing circuits, the operating clock frequency, the measured operating temperature, the queue utilization of the queue of computing circuits storing the allocated work blocks (information 126), and the application identifier specifying the type of application being processed by the corresponding computing circuit (and therefore the type of work block).

[0032] If scheduler 125 determines that there is a count of idle computing circuits equal to or greater than a threshold number of idle computing circuits, scheduler 125 allocates a certain number of work blocks equal to that threshold number to the idle computing circuits in computing circuits 124A-124B. In other words, scheduler 125 limits the number of idle computing circuits in computing circuits 124A-124B that can be activated at one time to that threshold number. In the example, device 100 has a total of 96 computing circuits, with 24 computing circuits in each of dies 122A-122B and 122C-122D. Die 122A has 16 idle computing circuits and 8 activated computing circuits that have already processed allocated work blocks. Scheduler 125's circuitry is capable of allocating 8 computing circuits simultaneously in a clock cycle, but for that particular clock cycle (or that particular scheduling window), the threshold number is 4. Therefore, scheduler 125 allocates work blocks to only 4 idle computing circuits instead of 8. This limitation on the number of computing circuits to be activated reduces the voltage drop in modules 110A-110B.

[0033] In some implementations, scheduler 125 modifies the scheduling rate. During a predetermined time period, scheduler 125 may limit the total number of idle computing circuits that are allocated work blocks and activated to process those work blocks. In one implementation, scheduler 125 defines this time period as a specific number of clock cycles. Scheduler 125 may determine to allocate work blocks to idle computing circuits at a rate of 2 idle computing circuits every clock cycle during this time period. At the end of this time period, scheduler 125 re-evaluates the scheduling based on updated parameters. For example, scheduler 125 redefines the threshold number of idle computing circuits based on updated values ​​of one or more of the following: the number of active computing circuits, the operating clock frequency, the measured operating temperature, the queue utilization of information 126, and any new application identifiers.

[0034] The queue utilization information 126 includes the number of available queue entries (or slots) in the queues of computing circuits 124A-124B. In one embodiment, these queues of computing circuits 124A-124B are implemented as first-in-first-out (FIFO) buffers. Each queue entry is capable of storing an allocated work block received from scheduler 142. Each queue entry may also be referred to as a "slot". The slot stores the program state of the allocated work block. In various embodiments, computing circuits 124A-124B maintains a count of the available slots or queue entries in the queues storing the allocated work blocks. Computing circuits 124A-124B sends this count as information 126 to scheduler 125. In some embodiments, when scheduler 125 allocates a certain number of work blocks (e.g., 8 work blocks) in a first scheduling window, computing circuits 124A-124B completes these 8 allocated work blocks before the start of a subsequent second scheduling window. However, in other implementations, computing circuits 124A-124B may not complete one or more of the eight assigned work blocks before the start of a subsequent second scheduling window. To prevent scheduling of more work blocks that unintentionally exceed the threshold number of idle computing circuits in a subsequent second scheduling window, scheduler 125 uses the updated queue utilization from information 126 to perform scheduling.

[0035] When a corresponding computing circuit in computing circuits 124A-124B begins execution of an assigned work block in one of its slots, the computing circuit uses a stored program state. In one embodiment, the stored program state includes a program counter and a pointer to a work item. If the instructions and data for the work item of the assigned work block are not yet available in the instruction cache and data cache of the computing circuit, the computing circuit uses the stored program state to retrieve the instructions and data for the work item. During a cache miss in one or more of the instruction cache and data cache, the delay in processing the cache miss may cause the assigned work block to remain in the slot at the start of the next scheduling window. Therefore, the assigned work unit has not yet completed execution. Depending on the delay in processing the cache miss, the assigned work block may not have started execution yet. The computing circuit sends the updated number of available slots (or other indication of queue utilization) as part of information 126 to scheduler 125. Scheduler 125 may assign 8 work blocks to computing circuit 124A, but when another scheduling window arrives, computing circuit 124A has only completed 6 of the 8 assigned work blocks, instead of completing all 8 assigned work blocks.

[0036] Without the aforementioned information (such as the updated number of available slots as part of information 126), scheduler 125 might unintentionally allocate more work blocks in subsequent scheduling windows than the threshold number of idle computing circuits. Using the aforementioned information, in one implementation, scheduler 125 will not allocate any further work blocks until all work blocks in the previously allocated work blocks have been completed. Therefore, scheduler 125 will not allocate any work blocks for one or more subsequent scheduling windows. In another implementation, scheduler 125 updates the threshold number of idle computing circuits for subsequent scheduling windows based on the received queue utilization of computing circuits 124A-124B provided in information 126. Each queue utilization (or queue utilization value) can be represented as one of the following: the number of available (unoccupied or idle) queue entries for the corresponding queue, the number of occupied (allocated with valid data) queue entries for the corresponding queue, the ratio of occupied queue entries to the total number of queue entries for the corresponding queue, the ratio of available queue entries to the total number of queue entries for the corresponding queue, etc.

[0037] In one embodiment, device 100 has a total of 64 computing circuits, with 16 computing circuits in each of dies 122A-122B and 122C-122D. For a first scheduling window, each of the 16 computing circuits is idle, and scheduler 125 determines the threshold number of idle computing circuits to be 8. Although there is a sufficient number of work blocks to allocate to all 16 computing circuits, scheduler 125 allocates 8 work blocks to die 122A instead of 16. This allocation of work blocks by scheduler 125 reduces the voltage drop of modules 110A-110B sharing the power rail.

[0038] During the subsequent second scheduling window, die 122A has 8 active compute circuits and 8 idle compute circuits. Scheduler 125 determines that the threshold number (or the threshold number of idle compute circuits) to allocate work blocks is 4. Therefore, scheduler 125 allocates 4 work blocks to the 4 compute circuits of die 122A instead of allocating 8 work blocks to the 8 compute circuits. This allocation of work blocks by scheduler 125 reduces the voltage drop of modules 110A-110B sharing the power rail. However, during execution, two of these four compute circuits have stopped executing work blocks due to one or more of an instruction cache miss and a data cache miss. Without information 126 regarding queue utilization, during the next (third) scheduling window, using the ratio of at least 4 idle compute circuits to 16 total compute circuits, scheduler 125 determines that the updated threshold number of idle compute circuits is 2. However, after instruction cache misses and / or data cache misses are processed, the two previously halted compute circuits can begin execution after the third scheduling window completes. Combining these two compute circuits (previously allocated during the second scheduling window) with the two compute circuits now receiving work blocks from scheduler 125 during the third scheduling window results in a total of four compute circuits that can begin execution. This number of four compute circuits that can begin execution after the third scheduling window completes exceeds the updated threshold of two for the number of idle compute circuits used in the third scheduling window.

[0039] To avoid the scenario described above where the updated threshold number of idle computing circuits for the third scheduling window exceeds 2, scheduler 125 uses the queue utilization information in information 126. Using the queue utilization information in information 126, during the third scheduling window, scheduler 125 knows that two of the previous four computing circuits assigned to the work block have stopped executing the work block, or otherwise did not start execution along with the arithmetic logic unit (ALU) and other computing circuits before the start time of the third scheduling window. Therefore, scheduler 125 determines that the updated threshold number of idle computing circuits is 0, not 2. Therefore, after the third scheduling is completed, the number of computing circuits that can start execution cannot exceed 2, and it is impossible to draw enough current to produce a voltage drop greater than the voltage threshold. No computing circuits may start execution after the third scheduling window because the two computing circuits (previously assigned during the second scheduling window) may remain stopped, or otherwise will not start execution before the start time of the subsequent fourth scheduling window.

[0040] Alternatively, in the third scheduling window, scheduler 125 determines the updated threshold number of idle computing circuits based on additional information (such as the number of computing circuits that have completed execution) within information 126. In one specific implementation, during the third scheduling window, scheduler 125 receives from information 126 an indication that three of the previously allocated computing circuits (allocated during the first scheduling window) have completed their allocated work blocks and that these three computing circuits will return to an idle state. Scheduler 125 also knows of two other computing circuits (allocated during the second scheduling window) that have stopped or otherwise simply performed memory accesses but have not yet started execution by the ALU before the start of the third scheduling window. Therefore, during this third scheduling window, scheduler 125 determines that the total number of idle computing circuits is the sum of four idle computing circuits that have not yet been allocated work blocks, two computing circuits that have stopped, and three computing circuits that have returned to an idle state from an active state. This totals nine idle computing circuits. The total of 9 idle computing circuits is greater than the number of idle computing circuits determined by 4 without considering information 126, which includes indications of 2 computing circuits that have stopped and 3 computing circuits that have returned from an active state to an idle state. Based on this total of 9 idle computing circuits, scheduler 125 determines a threshold number of idle computing circuits of 5. This threshold number of 5 is greater than the number of 4 idle computing circuits determined without considering information 126, and less than the number of 9 idle computing circuits determined according to usage information 126. Using information 126 to determine the threshold number of idle computing circuits for allocating work blocks allows scheduler 125 to allocate work blocks to computing circuits in a manner that reduces the voltage drop of modules 110A-110B on the shared power rail.

[0041] refer to Figure 2 Figure 200 shows a general block diagram illustrating the voltage drop of the power supply reference level of the integrated circuit. Figure 200 illustrates the power supply voltage 202 over time. Power supply voltage 202 is the power supply reference level on the power rail of the integrated circuit. During the operation of the integrated circuit, three different voltage drops 210, 212, and 214 occur over time. Simultaneous switching of the wide bus, when the corresponding power rail supplies a large amount of current to circuitry connected to that rail (such as a buffer of a wide bus), can cause a significant voltage drop in power supply voltage 202. This voltage drop (also referred to as the “voltage drop”) is expressed as ΔV, which is proportional to the expression L di / dt, where L is the parasitic inductance, and di / dt is the rate of change of current consumption from the power rail over time. In addition to the wide bus, near-simultaneous switching of numerous other nodes of the integrated circuit can also consume large amounts of current from the shared power rail. When such switching occurs, a significant voltage drop in power supply voltage 202 can occur.

[0042] As shown in the figure, voltage drop 210 occurs at time t0 (or time t0), voltage drop 212 occurs at time t1, power supply voltage 202 begins to drop again at time t2, and voltage drop 214 occurs at time t3. Voltage drop 210 occurs at time t0 when a large number of nodes switch simultaneously (e.g., a large number of idle computing circuits are activated simultaneously). In one specific implementation, power supply voltage 202 is 900 millivolts (mV), and voltage drop 210 is 300 mV. Nodes that previously stored Boolean logic high values ​​may be significantly degraded, causing the node voltage to appear as a Boolean logic low value to the circuit. Node voltage failures or even data loss may occur.

[0043] Once multiple idle computing circuits are activated, the power supply voltage 202 can temporarily return to its original value. However, due to the simultaneous switching of some wide buses and nodes when the application is processed by multiple active computing circuits, voltage drops 212 occur at time t1, and voltage drops 214 occur at time t3. These voltage drops 212 and 214 are not as large as voltage drop 210. External capacitors are placed between the power rails to create a passive bypass, which reduces power rail oscillations caused by external inductance. On-die capacitors are placed between the power rails on the semiconductor die to reduce internal inductance effects. These on-die capacitors are typically large to function, which consumes die area.

[0044] The computing circuitry also includes circuitry that performs reactive solutions to voltage drops, such as detecting and reducing the time rate of change of current consumption, di / dt. In some implementations, these reactive solutions include one or more of a temperature sensor and a current consumption measurement sensor. To be beneficial and to prevent voltage drops from lowering node voltages below a minimum threshold, these reactive solutions require initiating current reduction within a predetermined time window. Because voltage drop 210 occurs rapidly and initially when multiple idle computing circuits are activated, these capacitor and reactive circuitry solutions are not very effective at resisting this voltage drop. To reduce voltage drop 210, the scheduler actively limits the number of idle computing circuits that are allowed to be activated simultaneously.

[0045] In some implementations, when the scheduler actively limits the number of idle computing circuits allowed to be active simultaneously, the scheduler additionally uses the output measurements of temperature sensors and any voltage drop measurement circuitry. These circuits can provide non-zero voltage drop measurements. In one implementation, this type of information is included in (…). Figure 1Information 126 provides an indication of any initial or pre-existing voltage drop before and at time t0 in graph 200. For example, the integrated circuit has been executing the application for some time, and although the work block for the next iteration of the subroutine has not been allocated before time t0, the power supply voltage 202 is already lower than its expected value due to the current operating temperature and a pre-existing non-zero voltage drop. The scheduler uses this smaller power supply voltage value before and at time t0 when determining the threshold number of idle computing circuits and allocating work blocks to computing circuits.

[0046] Turn now Figure 3 The diagram shows a general block diagram of graph 300, which illustrates the number of active computing circuits of an integrated circuit over time. Graph 300 illustrates the number of active computing circuits over time. This number is also equal to the number of work blocks currently being executed, because each active computing circuit receives a work block. In some specific implementations, the computing circuits are SIMD circuits with multiple execution channels of the integrated circuit, and the work block is a wavefront (or wave) comprising multiple work items. Graph 300 includes waveforms 310 and 320. Waveform 310 illustrates the number of active computing circuits when the scheduler allocates work blocks to available computing circuits as quickly as possible. Waveform 320 illustrates the number of active computing circuits when the scheduler allocates work blocks in a manner that reduces voltage drop.

[0047] In the illustrated implementation, for the duration from time t0 to time t8, no activated computing circuit completes a work block and returns to an idle state. However, during other durations, such events occur. At time t0, no computing circuit is activated. Therefore, both waveforms 310 and 320 indicate that zero computing circuits are active. Between times t0 and t1, external circuits are retrieving and / or constructing work blocks for an application that has already begun running. The work blocks are sent to the scheduler. In some implementations, regardless of the scheduling algorithm implemented by the scheduler's circuitry, the rate of work block allocation and the rate of computing circuit activation are the same due to the initial rate of receiving work blocks when the application begins execution. In other implementations, the rates are different.

[0048] In various specific implementations, the scheduling algorithm corresponding to waveform 320 updates the rate of allocating work blocks and activating computing circuits at specific time intervals. This time interval is shown between each marked time in the marked time, such as at time t1, time t2, time t3, etc. This time interval may include a certain number of clock cycles based on the currently used operating clock frequency. Between time t1 and t2, waveform 310 indicates an increase in the rate of allocating work blocks and activating computing circuits. Waveform 310 indicates that the same higher rate is used without interruption until the maximum number of available computing circuits has been activated between time t4 and t5.

[0049] Compared to waveform 310, waveform 320 indicates that the same rate used between times t0 and t1 continues between times t1 and t2. At times t0 and t1, the scheduler corresponding to waveform 320 has determined that the count of pending work blocks to be allocated is less than the threshold number of idle computing circuits that can be allowed to be activated simultaneously. Again, if the number of work blocks initially sent to the scheduler is already high, the count of pending work blocks to be allocated may be equal to or greater than the threshold number of idle computing circuits, and the rate indicated by waveform 320 will decrease. At time t2, waveform 320 indicates an increase in the rate of work block allocation and computing circuit activation relative to the rate used between times t0 and t2. Again, at time t2, the scheduler corresponding to waveform 320 has determined that the count of pending work blocks to be allocated is less than the threshold number of idle computing circuits that can be activated simultaneously. However, the number of pending work blocks has increased, therefore the rate indicated by waveform 320 has increased.

[0050] At time t3, waveform 320 indicates a reduction in the rate at which work blocks are allocated and computing circuits are activated compared to the previously used rate. Here, the corresponding scheduler has determined that the count of idle computing circuits is equal to or greater than this threshold number. The scheduler allocates work blocks to only the threshold number of idle computing circuits, which reduces the rate at which work blocks are allocated and subsequently released. Due to the reduced rate of work block allocation and computing circuit activation, the rate of change of current consumption from the power rails (di / dt) decreases between times t3 and t4. Therefore, the resulting voltage drop is reduced between times t3 and t4. In some specific implementations, the rate of work block allocation and computing circuit activation is zero between times t3 and t4, so no additional idle computing circuits are activated. Therefore, the rate di / dt becomes zero between times t3 and t4.

[0051] It should be noted that in some specific implementations, the scheduler updates the threshold number of idle computing circuits that can be simultaneously activated over time based on one or more of the following: the number of pending work blocks, the number of activated computing circuits, the number of idle computing circuits, the queue utilization of the queue storing the allocated work blocks, the application identifier indicating the type of work to be performed by the activated computing circuits executing the work blocks, the power domain and / or P-state used by the activated computing circuits, etc. At times t4 and t6, waveform 320 indicates an increase in the rate of work block allocation and computing circuit activation, similar to the increase at time t2. At times t5 and t7, waveform 320 indicates a decrease in the rate of work block allocation and computing circuit activation, similar to the decrease at time t3.

[0052] Waveform 320 illustrates a reduction in the rate of work block allocation and simultaneous activation of computing circuitry to decrease voltage drop while maintaining a high level of performance. For example, the duration between the first time point in waveform 310 where the maximum number of computing circuitry is activated between times t4 and t5, and the second time point in waveform 320 where the maximum number of computing circuitry is activated between times t7 and t8, can be below a time threshold. Furthermore, it should be noted that the scheduler corresponding to waveform 320 actively reduces the rate of work block allocation and simultaneous activation of computing circuitry based on values ​​collected between these time intervals. For example, the reduction in the rate at time t3 is based on values ​​collected before time t3. Therefore, the scheduler uses an active method, not a passive one. Passive methods involve on-die measurement of current consumption and limiting one or more of the work block allocation at the scheduler or the issuance of commands at the activated computing circuitry based on whether the measurement exceeds a threshold. It is possible and conceivable to use active methods in conjunction with passive methods in integrated circuits to reduce voltage drop.

[0053] Turn now Figure 4 This diagram illustrates a general block diagram of a method 400 for efficiently managing voltage drops in multiple computing circuits of an integrated circuit. For illustrative purposes, the steps in this embodiment are shown in a sequential order. However, in other embodiments, some steps occur in a different order than shown, some steps are performed concurrently, some steps are combined with other steps, and some steps are absent.

[0054] The integrated circuit includes multiple replica computing circuits. A scheduler allocates work blocks to idle computing circuits, which are then activated and process the allocated work blocks. The scheduler detects that a scheduling window has begun (box 402). For example, the scheduler determines that a specific time period has elapsed since the most recent allocation of a work block was executed. The scheduler determines a threshold number of idle computing circuits that can be activated simultaneously (box 404). In some implementations, the scheduler determines this threshold number of idle computing circuits based on one or more of the following: the number of active computing circuits with allocated work blocks (or conversely, the number of idle computing units without allocated work blocks), the total number of computing circuits, the number of stopped computing circuits, the number of computing circuits that have recently completed executing a work block, the number of work blocks pending processing, any measured pre-existing voltage drop, the operating clock frequency, the measured operating temperature, the queue utilization of the queue storing the allocated work blocks, and the application identifier.

[0055] In one implementation, the scheduler determines a threshold number of idle computing circuits that can be activated simultaneously based on the ratio of the number of idle computing circuits to the total number of computing circuits. To determine the number of idle computing circuits, the scheduler includes a certain number of computing circuits that have been previously assigned work blocks but have not yet begun executing those work blocks. To determine the number of idle computing circuits, the scheduler also includes a certain number of computing circuits that have executed work blocks but have completed their execution. In some implementations, when the scheduler receives an indication of a pre-existing measurement of a non-zero voltage drop, the scheduler reduces the threshold number of idle computing circuits that can be activated simultaneously.

[0056] If the scheduler determines that there is a count of idle computing circuits equal to or greater than the threshold number of idle computing circuits (the "Yes" branch of condition box 406), the scheduler allocates a certain number of work blocks equal to the threshold number to the idle computing circuits (box 408). In other words, the scheduler limits the number of idle computing circuits that can be activated at one time to the threshold number. In various specific implementations, the scheduler limits the number of idle computing circuits that can be activated simultaneously, even when it receives an indication of a power performance status (indicating that one of the pending work blocks should be allocated to each of the idle computing circuits to maintain a certain throughput). In the example, the integrated circuit has a total of 24 computing circuits, of which 16 computing circuits are idle and 8 computing circuits are activated and have processed their allocated work blocks. The scheduler's circuitry is capable of allocating 8 computing circuits simultaneously in a clock cycle, but for that particular clock cycle, the threshold number is 4. As previously described, the scheduler determines the threshold number of idle computing circuits based on one or more of the following: the number of active computing circuits, the total number of computing circuits, the number of stopped computing circuits, the number of computing circuits that have recently completed executing a work block, the number of pending work blocks, any measured pre-existing voltage drop, the operating clock frequency, the measured operating temperature, the queue utilization of the queue storing the allocated work blocks, and the application identifier. Therefore, the scheduler allocates work blocks to only 4 idle computing circuits instead of 8.

[0057] In some implementations, the scheduler changes the scheduling rate. During a predetermined time period, the scheduler limit may be the total number of idle computing circuits that are allocated work blocks and activated to process those work blocks. In one implementation, the scheduler defines this time period as a specific number of clock cycles. The scheduler may determine to allocate work blocks to idle computing circuits at a rate of 2 idle computing circuits every clock cycle during this time period. At the end of this time period, the scheduler re-evaluates the scheduling based on updated parameters. For example, the scheduler redefines the threshold number of idle computing circuits based on updated values ​​of: the number of active computing circuits, the operating clock frequency, the measured operating temperature, the queue utilization of the queue storing the allocated work blocks, and any new application identifiers. The scheduler repeats the steps in blocks 402 to 406 of method 400.

[0058] If the scheduler determines that there is a count of idle computing circuits less than the threshold number (the "No" branch of condition block 406), the scheduler allocates a number of work blocks less than or equal to that count to the idle computing circuits (block 410). If there is no data dependency between the work blocks to be allocated and the work blocks that have already been executed, the scheduler allocates a number of work blocks equal to the count of the idle computing circuits to the idle computing circuits. Otherwise, the scheduler allocates a number of work blocks less than the count of the idle computing circuits to the idle computing circuits. The integrated circuit activates the idle computing circuit that has been allocated work blocks (block 412). The activated computing circuit executes the allocated work blocks (block 414).

[0059] Turn now Figure 5 This diagram illustrates a general block diagram of a computing system 500 that efficiently manages voltage drops across multiple computing circuits of an integrated circuit. As shown, the computing system 500 includes a processor 510, a memory 520, and a parallel data processor 530. In some embodiments, the functionality of the computing system 500 is incorporated as a component on a single die (such as a single integrated circuit). In other embodiments, the functionality of the computing system 500 is incorporated as multiple dies on a system-on-a-chip (SOC). In still other embodiments, the functionality of the computing system 500 is implemented by multiple separate dies that have been fabricated on separate silicon wafers and placed in a system package called a multi-chip module (MCM). In various embodiments, the computing system 500 is used in desktop computers, portable computers, mobile devices, servers, peripherals, etc.

[0060] Processor 510's circuitry processes instructions of a predetermined algorithm. This processing includes fetching instructions and data, decoding instructions, executing instructions, and storing results. In one embodiment, processor 510 uses one or more processor cores with circuitry for executing instructions according to a predefined general-purpose instruction set architecture (ISA). In various embodiments, processor 510 is a general-purpose central processing unit (CPU). Parallel data processor 530 uses one or more processor cores with a relatively wide single-instruction multiple-data (SIMD) microarchitecture to achieve high throughput in highly data-parallel applications. In one embodiment, parallel data processor 530 is a graphics processing unit (GPU). In other embodiments, parallel data processor 530 is another type of circuitry.

[0061] In various implementations, the computing circuit 534 is a SIMD circuit with multiple execution channels. The scheduler 532 schedules work blocks to the computing circuit 534 in a manner that minimizes voltage drop across the computing circuit 534. In various implementations, the scheduler 532 includes ( Figure 1The scheduler 125 (or scheduler 127 or scheduler 142) performs the described scheduling steps to generate ( Figure 3 The waveform of curve 300 is shown in curve 320. This type of function reduces voltage drop, such as (…). Figure 2 The voltage drops by 210. Here, a work block is a wavefront (or wave) of multiple work items. The parallel data processor 530 is efficient for parallel computation of data found within the loop of an application (such as in applications used for manipulating, rendering, and displaying computer graphics). In such cases, each data item in the wavefront data items is a pixel of an image. These applications may also include molecular dynamics simulations, financial calculations, neural network training, etc. The highly parallel architecture of the parallel data processor 530 makes it more efficient than the general-purpose architecture of the processor 510.

[0062] In various implementations, threads are scheduled on one of the processors 510 and 530 in a manner that maximizes instruction throughput per thread, at least in part based on the runtime hardware resources of the processor 510 and the parallel data processor 530. In some implementations, some threads are associated with general-purpose algorithms scheduled on the processor 510, while others are associated with parallel data computation-intensive algorithms (such as video graphics rendering algorithms) scheduled on the parallel data processor 530. Computational circuitry 534 can be used for real-time data processing, such as rendering multiple pixels, image blending, pixel shading, vertex shading, and geometry shading. Some threads that are not video graphics rendering algorithms still exhibit parallel data and intensive throughput. These threads have instructions capable of operating on a relatively large number of different data elements (or data items) simultaneously. Examples of such threads are those used for scientific, medical, financial, and cryptographic / decryption computations.

[0063] To change the scheduling of the aforementioned computations from processor 510 to parallel data processor 530, a software development kit (SDK) and application programming interface (API) were developed for use with widely available high-level languages, providing supporting function calls. Function calls provide an abstraction layer of the parallel implementation details of parallel data processor 530. These details are hardware-specific to parallel data processor 530 but are hidden from developers to allow for more flexible writing of software applications. Function calls in high-level languages ​​(such as C, C++, FORTRAN, and Java) are translated into commands, which are later processed by the hardware in parallel data processor 530. Although a network interface is not shown, in some implementations, parallel data processor 530 is used by remote programmers in a cloud computing environment.

[0064] The software application begins execution on processor 510. Function calls within the application are translated into commands by a given API. Processor 510 sends the translated commands to memory 520 for storage in a ring buffer 522. Commands are placed in groups called command groups. In some implementations, processors 510 and 530 use a producer-consumer relationship, also known as a client-server relationship. Processor 510 writes commands to ring buffer 522. Circuitry of the controller (not shown) of parallel data processor 530 reads commands from ring buffer 522. In some implementations, this controller is the command processor of the GPU. The controller sends work blocks to scheduler 532, which allocates work blocks to computation circuitry 534. By doing so, parallel data processor 530 processes the commands and writes the resulting data to buffer 524. Processor 510 is configured to update the write pointer of ring buffer 522 and provide the size of each command group. The parallel data processor 530 updates the read pointer of the ring buffer 522 and indicates the entry that the next read operation in the ring buffer 522 will use.

[0065] refer to Figure 6 This diagram illustrates a general block diagram of an integrated circuit 600 that efficiently manages voltage drops across multiple computing circuits. In the illustrated embodiment, the integrated circuit 600 includes two partitions, such as partition 610 and partition 650. Each of partitions 610 and 650 includes components for processing work blocks. Partition 610 includes a cache memory 620 accessed by computing resource 630. Computing resource 630 includes computing circuits 640A, 640B, and 640C (or computing circuits 640A-640C). Partition 650 includes clients 660-662. Control block 670 includes a scheduler 672 and a power manager 674. In some embodiments, the scheduler 672 and power manager 674 have ( Figure 1 The scheduler 142 and power manager 144 have the same functions. For example, in one embodiment, the scheduler 672 of control block 670 schedules the working blocks on computing circuits 640A-640C of partition 610. In other embodiments, the scheduler 622 of partition 610 schedules the working blocks on computing circuits 640A-640C. Either scheduler 672 or scheduler 622 schedules the working blocks on computing circuits 640A-640C in a manner that reduces voltage drop on computing circuits 640A-640C. In various embodiments, scheduler 672 (or scheduler 622) includes scheduler 125 (or... Figure 1 The scheduler 127 or scheduler 142) functions and performs the described scheduling steps to generate ( Figure 3 The waveform of curve 300 is shown in curve 320. This type of function reduces voltage drop, such as (…). Figure 2 The voltage dropped by 210.

[0066] For ease of illustration, communication structures, memory controllers, interrupt controllers, and phase-locked loops (PLLs) or other clock generation circuits are not shown. In some embodiments, the functionality of integrated circuit 600 is incorporated as a component on a single die (such as a single integrated circuit). In one embodiment, the functionality of device 100 is incorporated as one of a plurality of dies on a system-on-a-chip (SOC). In other embodiments, the functionality of integrated circuit 600 is implemented by a plurality of separate dies that have been fabricated on separate silicon wafers and placed in a system package called a multi-chip module (MCM). In various embodiments, device 100 is used in desktop computers, portable computers, mobile devices, servers, peripherals, etc.

[0067] In some embodiments, each of partitions 610 and 650 is assigned to a corresponding power domain. In other embodiments, each of partitions 610 and 650 is assigned to the same power domain. A power domain includes at least operating parameters, such as at least an operating supply voltage and an operating clock frequency. A power domain also includes control signals for enabling and disabling connections to a clock generation circuit and one or more power supply references. In information 682, partition 610 receives operating parameters for a first power domain from power controller 670. In information 684, partition 650 receives operating parameters for a second power domain from power controller 670.

[0068] Clients 660-662 include various types of circuitry, such as central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), multimedia engines, etc. Each of clients 660-662 is capable of handling work blocks of various workloads. In some specific implementations, work blocks scheduled on partition 610 include wavefronts, while work blocks scheduled on partition 650 include instructions that operate on individual data items and are not grouped into wavefronts. Furthermore, each of clients 660-662 is capable of generating and processing one or more requests of various types, such as memory access read and write requests and cache snooping requests.

[0069] In one embodiment, integrated circuit 600 is a graphics processing unit (GPU). The computing resources 630 of partition 610 process highly data-parallel applications. Computing resources 630 include multiple computing circuits 640A-640C, each with multiple channels 642. In some embodiments, channels 642 operate in a lockstep manner. In various embodiments, the data flow within each channel of channel 642 is pipelined. Pipeline registers are used to store intermediate results, and circuits for the arithmetic logic unit (ALU) perform integer arithmetic, floating-point arithmetic, Boolean logic operations, branch condition comparisons, etc. These components are not shown for illustration. Each computing circuit in a given row across channel 642 is the same computing circuit. Each of these computing circuits operates on the same instructions, but different data is associated with different threads. As previously described, a number of work items are grouped into wavefronts for simultaneous execution by multiple SIMD execution channels (such as channels 642 of computing circuits 640A-640C). Each data item is processed independently of other data items, but uses the same sequence of operations from the subroutine.

[0070] As shown in the figure, each of the computing circuits 640A-640C also includes a corresponding queue 643, register file 644, local data repository 646, and local cache memory 648 for storing allocated work blocks. In some embodiments, the local data repository 646 is shared among the channels 642 within each of the computing circuits 640A-640C. In other embodiments, the local data repository is shared among the computing circuits 640A-640C. Therefore, one or more channels of the channels 642 within the computing unit 640A may share result data with one or more channels 642 within the computing circuit 640A based on the operating mode. In one embodiment, the queue 643 is implemented as a first-in, first-out (FIFO) buffer. Each queue entry of the queue 643 is capable of storing the allocated work block received from the scheduler 622 (or scheduler 672). Each queue entry may also be referred to as a "slot". The slot stores the program state of the allocated work block. In various specific implementations, computing circuits 640A-640C maintain a count of available slots or queue entries in the queue storing the allocated work blocks. Computing circuits 640A-640C send this count as information to scheduler 622 (or scheduler 672). Although an example of a Single Instruction Multiple Data (SIMD) microarchitecture is shown for computing resource 630, other types of highly parallel data microarchitectures are possible and conceivable. The high parallelism provided by the hardware of computing resource 630 is used for rendering multiple pixels simultaneously, but it is also capable of processing multiple data elements simultaneously for scientific, medical, financial, encryption / decryption, and other operations.

[0071] Clients 660-662 may also include one or more of an analog-to-digital converter (ADC), a scan converter, a video decoder, a display controller, and other computing circuitry. In some implementations, partition 660 is used for real-time data processing, while partition 650 is used for non-real-time data processing. Examples of real-time data processing include rendering multiple pixels, image blending, pixel shading, vertex shading, and geometry shading. Examples of non-real-time data processing are multimedia playback, such as video decoding of encoded audio / video streams, image scaling, image rotation, color space conversion, power-on initialization, background processes (such as garbage collection), etc. Circuitry of the controller (not shown) receives tasks. In some implementations, the controller is the GPU's command processor, and the task is a sequence of commands (instructions) for function calls from the application. The controller assigns the task to one of the two partitions 610 and 650 based on the task type of the received task. One of the schedulers 672 and 622 receives these tasks from the controller, organizes them into work blocks (if not already done), and schedules the work blocks on the computing circuits 640A-640C in a manner that minimizes voltage drop on the computing circuits 640A-640C.

[0072] Turn now Figure 7 A general block diagram of a scheduler 700 for managing voltage drops in a replica computing circuit of an integrated circuit is shown. As shown, the scheduler 700 includes a table 710 and control circuitry 730. Control circuitry 730 receives input value 740 and generates work block allocations 750 for multiple replica computing circuits of the integrated circuit. Control circuitry 730 generates work allocations 750 in a manner that reduces voltage drops on one or more power rails used by the multiple replica computing circuits. Control circuitry 730 includes components 732-738 for allocating work blocks to the multiple replica computing circuits. In some embodiments, the computing circuitry is a single instruction multiple data (SIMD) circuit including multiple execution channels, and the work block is a wavefront including multiple work items. Table 710 includes multiple table entries (or more entries), each storing information in multiple fields (such as at least fields 712-722). In some embodiments, one or more components and corresponding functions of the scheduler 700 are provided in another external circuit (not here in the scheduler 700). In various embodiments, also ( Figure 1 Scheduler 125 (or scheduler 127 or scheduler 142), ( Figure 5 Scheduler 532, ( Figure 6 Scheduler 672 (or scheduler 622) and ( Figure 8 The scheduler 812 provides the functionality provided by the scheduler 700.

[0073] Table 710 is implemented using one of the following: a flip-flop circuit, random access memory (RAM), or content-addressable memory (CAM). While specific information is shown stored in fields 712-722 in a specific sequential order, different orders are used in other implementations, and different amounts and types of information are stored. Field 712 stores status information, such as at least a valid bit and an indication of whether the corresponding computing circuit is active or idle. An active computing circuit uses the assigned P-state operating parameters to process the assigned work block. An idle computing circuit does not have an assigned work block and is typically clock-gated, or the idle computing circuit has disabled the connection to the clock generation circuit. Field 714 stores the identifier of one of a plurality of replica modules of the specified integrated circuit. Field 716 stores the identifier of one of a plurality of dies within the specified module. Field 718 stores the identifier of one of a plurality of computing circuits within the specified die.

[0074] Field 720 stores operating parameters for a specific power performance state (P-state) currently used by the corresponding computing circuit, such as at least the operating supply voltage and operating clock frequency. In one embodiment, one or more of the operating supply voltage and operating clock frequency are used by control circuitry 730 to generate work block allocation 750. Field 722 stores other reported information, such as an application identifier specifying the type of application being processed by the corresponding computing circuit (and therefore the type of work block). The type of application provides an indication of the activity level or power consumption required for the work block processing that application. Another example of information in field 722 is whether a first work block of the corresponding computing circuit is dependent on data from a second work block, and therefore both the first and second work blocks should be allocated to the corresponding computing circuit. Field 722 may also store an indication of the power rails in cases where some computing circuits of the integrated circuit do not use the same power rails as other computing circuits.

[0075] Control circuitry 730 receives input value 740, which is used in conjunction with values ​​stored in table 710 and configuration register 734 to generate work block allocation 750. Although a specific number and type of input values ​​are shown, in other embodiments, control circuitry 730 uses a different number and type of input values. Input value 740 includes a time period count 742, a time period count that is a timer output value that counts the number of clock cycles since control circuitry 730 last generated work block allocation 750. In another embodiment, configuration register 734 includes a timer that maintains this count value. Control circuitry 730 compares time period count 742 with a threshold time period 738, and when a match occurs, work block allocation selector 732 uses values ​​stored in table 710 and configuration register 734 to generate work block allocation 750. In various embodiments, a match occurs when a threshold time period 738 has elapsed since the most recent allocation 750 from a work block to an idle computing circuit.

[0076] In one embodiment, the time period count 742 is a clock cycle count. In one embodiment, when the control circuit has generated a work block allocation 750, one or more of the external circuitry and the control circuitry 730 resets the time period count 742. In one embodiment, the time period indicated by the threshold time period 738 is less than the duration for which the computing circuitry completes execution of the allocated work block. In other embodiments, the time period indicated by the threshold time period 738 is greater than the duration for which the computing circuitry completes execution of the allocated work block. In some embodiments, the control circuitry 730 maintains multiple values ​​up to the threshold time period 738 and selects a specific value based on one or more of the input value 740 and values ​​stored in table 710. For example, the value of the threshold time period 738 for a high-performance and high-power application running on a computing circuitry using high-performance P-state operating parameters may differ from the value of the threshold time period 738 for a low-performance and low-power application running on a computing circuitry using low-performance P-state operating parameters.

[0077] In some embodiments, control circuitry 730 determines from table 710 the count 744 of idle computing circuits on the same voltage rail and the number 746 of idle computing circuits on individual voltage rails. In some embodiments, the number (count) 744 of idle computing circuits on the same voltage rail is based on the queue utilization of the queue of computing circuits that control circuitry 730 receives and stores allocated work blocks. In other embodiments, when a computing circuit becomes idle, one or more configuration registers in configuration register 734 (such as the count 734 of computing circuits and table 710) are updated, and one or more of the input values ​​744 and 746 are adjusted. When available computing circuits use two or more enabled power reference level rails (or power rails) (power switches connect the power rails to the computing circuits), scheduler 700 can allocate work blocks across different power rails among available computing circuits. Distributing work blocks in this way helps reduce voltage drop on any one power rail. In such cases, configuration register 734 may include a separate set of registers for values ​​736-738 for each power rail.

[0078] Input value 746 is not used when each available computing circuit in the available computing circuitry uses a single shared voltage rail. In such cases, configuration register 734 includes a single set of registers for values ​​736-738 for a single available power rail. In one embodiment, the number 748 work blocks to be processed is provided by external circuitry. In another embodiment, control circuitry 730 determines input value 748 based on one or more applications being processed. An incoming application identifier 749 specifies the type of application (and therefore the type of work block) to be processed by one or more computing circuits. The type of application provides an indication of the activity level or power consumption required to process the work block of the incoming application.

[0079] In some implementations, control circuitry 730 also receives an indication of the operating temperature. In such implementations, the integrated circuit provides a measured temperature value derived from measurements by analog and / or digital thermal sensors placed throughout the integrated circuit. When time period count 742 indicates that the next allocation of a work block should be performed (such as matching a threshold time period 738), the hardware of work block allocation selector 732 (or selector 732) generates a work block allocation 750 based at least on input value 740 and values ​​stored in table 710. Selector 732 allocates work blocks to a smaller number of available idle computing circuits, rather than allocating work blocks to the maximum number of available idle computing circuits to maximize performance. This smaller number is based on reducing the voltage drop across the power rails used by multiple computing circuits. This smaller number is represented by the threshold number 737 of idle computing circuits in configuration register 734.

[0080] In some implementations, the threshold number 737 of the idle computing circuit is based on one or more of the input value 740 and values ​​stored in table 710, and this number is determined during the testing and characterization of the integrated circuit. Each of these threshold numbers of the idle computing circuit provides a voltage drop of a corresponding power supply reference level used by multiple computing circuits, which is less than a voltage threshold. Therefore, one or more threshold numbers of the idle computing circuits simultaneously activated at a specific time are stored in one or more of the following: a table (separate from table 710), another data structure, a memory storing firmware, and ROM (or EPROM). Selector 732 generates an index based on one or more of the input value 740 and values ​​stored in table 710, and then uses the generated index to index into the table or other data structure and retrieve the threshold number of the idle computing circuit to be activated. Selector 732 stores the retrieved value in configuration register 734 in a programmable register of configuration register 734 as the threshold number 737 of the idle computing circuit. In another specific implementation, the control circuit 700 locally stores a data structure containing a threshold number 737 of idle calculation circuits and retrieves the value to be allocated based on a condition satisfied by one or more of the input value 740 and the values ​​stored in table 710.

[0081] In some implementations, control circuitry 730 maintains a count 736 of idle computing circuits among multiple computing circuits sharing a power reference level rail (power rail) in configuration register 734. To maintain this count 736, selector 732 uses table 710 and any received updates used to update the active / idle status information of table 710. When selector 732 determines that time period count 742 matches (or is no longer less than) a threshold time period 738, selector 732 ensures that the number of idle computing circuits to be allocated work blocks and activated does not exceed a threshold number 737 of idle computing circuits. For example, when selector 732 determines that time period count 742 matches (or is no longer less than) a threshold time period 738, and selector 732 determines that the count 734 of idle computing circuits is equal to or greater than the threshold number 737 of idle computing circuits, the selector allocates a threshold number of work blocks to a certain number of idle computing circuits that is less than or equal to the threshold number 737 of idle computing circuits. The scheduler 700 or other external circuitry activates the allocated idle computing circuits and provides these circuits with allocated work blocks for processing. When the first work block of a corresponding computing circuit is data dependent on the second work block, and therefore both the first and second work blocks should be allocated to the corresponding computing circuit, the selector 732 may allocate a certain number of idle computing circuits less than the threshold number 737 of idle computing circuits.

[0082] In the example, the integrated circuit includes 12 computing circuits, and the count 744 of idle computing circuits on the same voltage rail is 9 idle computing circuits. Based on the operating conditions determined from the input value 740 and the values ​​stored in table 710, the control circuit 730 determines that the threshold number 737 of idle computing circuits is 6 idle computing circuits. The number of work blocks to be processed 748 is 18. When the selector 732 determines that the time period count 742 matches (or is no longer less than) the threshold time period 738, and the selector 732 determines that the count 734 of idle computing circuits (9 computing circuits) is equal to or greater than the threshold number 737 of idle computing circuits (6 computing circuits), the selector allocates the work block to the idle computing circuits. However, the scheduler 700 allocates 6 computing circuits (i.e., the threshold number 737 of idle computing circuits) to 6 work blocks instead of allocating 9 idle computing circuits (i.e., the maximum number) to 9 work blocks. Therefore, the scheduler 700 reduces the voltage drop on the power rail shared by the idle computing circuits. For example, the scheduler 700 provides a voltage drop across the power rails used by 12 computing circuits that is less than a voltage threshold.

[0083] Data dependencies between work blocks may cause the scheduler 700 to allocate only 4 or 5 work blocks to 4 or 5 idle computing circuits, which is less than the threshold number of idle computing circuits 737 (i.e., 6 idle computing circuits) and the maximum number (i.e., 9 idle computing circuits). When the number of work blocks to be processed 748 is 5 (which is less than the threshold number of idle computing circuits 737 (i.e., 6 idle computing circuits)), the scheduler 700 is able to allocate all 5 work blocks to be processed to 5 idle computing circuits without the data dependencies reducing the number of allocated idle computing circuits. In some specific implementations, the integrated circuit is a graphics processing unit (GPU) or another parallel data processing circuit.

[0084] In the example, the integrated circuit includes 8 chiplets, each chiplet containing 6 computing circuits, each computing circuit containing 2 SIMD circuits, and each SIMD circuit containing 64 execution channels to execute 64 work items simultaneously. Therefore, the integrated circuit is able to execute 6,144 work items simultaneously across 6,144 execution channels because there are 6,144 available execution channels (64 × 2 × 6 × 8 = 6,144). A wavefront (or wave) is 64 work items, and the scheduler 700 treats the wavefront as a work block. Therefore, in this example, the scheduler 700 treats the SIMD circuits as computing circuits, and since there are 96 computing circuits (2 × 6 × 8 = 96), there are a maximum of 96 work blocks that can be executed simultaneously on 96 computing circuits. For Table 710, the scheduler 700 also treats the chiplets as dies and the computing circuits as modules. The values ​​736-738 stored in the configuration register 734 for this example are updated based on one or more of the input value 740 and the values ​​stored in table 710.

[0085] Turn now Figure 8 This diagram illustrates a generalized block diagram of a system-in-package (SiP) 800 that manages voltage drops across multiple computing circuits of an integrated circuit. In various embodiments, a three-dimensional (3D) package is used within the computing system. This type of package is referred to as a system-in-package (SiP). A SiP comprises one or more three-dimensional integrated circuits (3D ICs). A 3D IC comprises two or more layers of active electronics integrated vertically and / or horizontally into a single circuit. In one embodiment, interposer-based integration is used, thereby placing the 3D IC next to the processor 810. Alternatively, the 3D IC is stacked directly on top of another IC.

[0086] Die stacking technology is a manufacturing process that enables the physical stacking of multiple individual silicon dies (integrated chips) together in the same package, providing high-bandwidth and low-latency interconnects. In some implementations, dies are stacked side-by-side on a silicon interposer or directly stacked vertically on top of each other. One configuration of SiP is to stack one or more semiconductor dies (or dies) next to and / or on top of a processor (such as processor 810). In one implementation, SiP 800 includes processor 810 and modules 840A-840B. Module 840A includes semiconductor die 820A and multiple three-dimensional (3D) semiconductor dies 822A-822B within partition 850A. Although two dies are shown, in other implementations, any number of dies are used as stacked 3D dies.

[0087] Similarly, module 840B includes semiconductor die 820B and a plurality of 3D semiconductor dies 822C-822D within partition 850B. Although not shown, each of dies 822A-822B and dies 822C-822D includes one or more computing circuits. Scheduler 812 schedules working blocks on these computing circuits in a manner that reduces voltage drops on the computing circuits within dies 822A-822B and 822C-822D. In various specific embodiments, scheduler 812 includes ( Figure 1 Scheduler 125 (or scheduler 127 or scheduler 142), ( Figure 5 Scheduler 532, ( Figure 6 Scheduler 672 (or scheduler 622) and ( Figure 7 The scheduler 700 performs the functions of the scheduler. The scheduler 812 executes the described scheduling steps to generate (the scheduler). Figure 3 The waveform of curve 300 is shown in curve 320. This type of function reduces voltage drop, such as (…). Figure 2 The voltage dropped by 210.

[0088] Dies 822A-822B within partition 850A share at least the same power rails. Dies 822A-822B may also share the same clock signal. Dies 822C-822D within partition 850B share at least the same power rails. Dies 822C-822D may also share the same clock signal. In some embodiments, another module is placed adjacent to the left side of module 840A, and this other module includes a die as an instantiated copy of die 820A. It is possible and conceivable for this other die to share at least the power rails with die 820A.

[0089] Each of modules 840A-840B communicates with processor 810 via a horizontal low-latency interconnect 830. In various implementations, processor 810 is a general-purpose central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), field-programmable gate array (FPGA), or other data processing device. When not using a SiP, the in-package horizontal low-latency interconnect 830 shortens the length of the interconnect signals compared to long off-chip interconnects. The in-package horizontal low-latency interconnect 830 uses specific signals and protocols, just as chips such as processor 810 and modules 840A-840B are mounted in separate packages on a circuit board. In some implementations, SiP 800 additionally includes a back-side via or through-body silicon via 832 leading to external package connectors 834. External package connectors 834 are used for input / output (I / O) signals and power signals.

[0090] In various implementations, multiple device layers are stacked on top of each other, with direct vertical interconnects 836 tunneling through these device layers. In various implementations, the vertical interconnects 836 are multiple through-silicon vias (TSBs) grouped together to form through-silicon buses (TSBs). TSBs serve as vertical electrical connections traversing the silicon wafer. TSBs are an alternative interconnect for wire bonding and flip-chip applications. The size and density of the vertical interconnects 836 that tunnel between different device layers vary based on the underlying technology used to manufacture 3DICs.

[0091] As shown in the figure, some of the vertical interconnects in vertical interconnect 836 do not traverse each of modules 840A-840B. Therefore, in some embodiments, processor 810 does not have direct connections to one or more dies (such as die 822D in the illustrated embodiment). Thus, information routing depends on the other dies of SiP 800. In various embodiments, dies 822A-822B and 822C-822D are chiplets. As used herein, a “chiplet” is also referred to as a “block of intellectual property” (or IP block). However, a “chiplet” is a semiconductor die (or die) that is manufactured separately from other dies within a single integrated circuit in an MCM and then interconnected with those other dies.

[0092] On a single silicon wafer, only a plurality of chiplets are manufactured as multiple instantiated copies of a particular integrated circuit, rather than together with other computing circuitry that does not use instantiated copies of the particular integrated circuit. For example, the chiplets are not manufactured on the silicon wafer together with various other computing circuitry on a larger semiconductor die such as a SoC. A first silicon wafer (or first wafer) is manufactured using multiple instantiated copies of the integrated circuit of the first chiplet, and the first wafer is cut using laser dicing techniques to separate the multiple copies of the first chiplet.

[0093] A second silicon wafer (or second dies) is fabricated using multiple instantiated copies of the integrated circuit of the second chiplet, and the second wafer is diced using laser dicing technology to separate the multiple copies of the second chiplet. The first chiplet provides a function different from that of the second chiplet. One or more copies of the first chiplet are placed in an integrated circuit, and one or more copies of the second chiplet are placed in an integrated circuit. The first and second chipslets are interconnected within their respective MCMs. This process replaces the process of fabricating a third silicon wafer (or third dies) using multiple copies of a single monolithic semiconductor die, which includes the functions of the first and second chiplets as integrated computing circuits within the single monolithic semiconductor die.

[0094] The process yield of a single monolithic die on a silicon wafer is lower than that of a smaller chiplet on a separate silicon wafer. Furthermore, semiconductor processes are adaptable to specific types of chiplets being manufactured. For a single monolithic die, each die on the wafer is formed using the same manufacturing process. However, interface computing circuits may not require the process parameters of expensive processes used by semiconductor manufacturers that enable the fastest and smallest geometry, which is beneficial for high-throughput circuitry on a die. By utilizing individual chiplets, designers can add or remove chiplets for specific integrated circuits, easily producing products with a wide range of performance categories. In contrast, when using a single monolithic die, an entire new silicon wafer must be manufactured for different products. Figure 1 (The) bare film 122A-122D and ( Figure 6 It is possible and conceivable that the computing resources 630 or partition 610 could also be implemented as chiplets.

[0095] It should be noted that one or more of the above-described embodiments include software. In such embodiments, program instructions for implementing the method and / or mechanism are transmitted or stored on a computer-readable medium. Many types of media configured to store program instructions are available and include hard disks, floppy disks, CD-ROMs, DVDs, flash memory, programmable ROMs (PROMs), random access memory (RAMs), and various other forms of volatile or non-volatile storage devices. Generally, computer-accessible storage media include any storage medium that can be accessed by a computer during use to provide instructions and / or data to the computer. For example, computer-accessible storage media include storage media such as magnetic or optical media, such as magnetic disks (fixed or removable), magnetic tape, CD-ROMs or DVD-ROMs, CD-Rs, CD-RWs, DVD-Rs, DVD-RWs, or Blu-ray discs. Storage media also include volatile or non-volatile storage media, such as RAM (e.g., Synchronous Dynamic RAM (SDRAM), Dual Data Rate (DDR, DDR2, DDR3, etc.) SDRAM, Low Power DDR (LPDDR2, etc.) SDRAM, Rambus DRAM (RDRAM), Static RAM (SRAM), etc.), ROM accessible via a peripheral device interface (such as a Universal Serial Bus (USB) interface), flash memory, and non-volatile memory (e.g., flash memory). Storage media include microelectromechanical systems (MEMS), and storage media accessible via communication media such as networks and / or wireless links.

[0096] Additionally, in various specific implementations, program instructions include behavioral-level or register-transfer-level (RTL) descriptions of hardware functionality in high-level programming languages ​​(such as C) or design languages ​​(HDLs) (such as Verilog, VHDL, or database formats (such as GDS II streaming format (GDSII)). In some cases, the description is read by a synthesis tool that synthesizes the description to produce a netlist comprising a list of gates from a synthesis library. The netlist includes gate sets, which also represent the functionality of the hardware comprising the system. The netlist is then placed and strung to produce a dataset describing the geometry to be applied to a mask. The mask is then used in various semiconductor manufacturing steps to produce semiconductor circuits or circuits corresponding to the system. Alternatively, computer-accessible instructions on a storage medium are, as desired, netlists (with or without synthesis libraries) or datasets. Furthermore, the instructions are used by, for example... and Mentor The purpose of this type of supplier's hardware-based type simulator is to perform simulation.

[0097] Although the specific embodiments described above have been described in considerable detail, many variations and modifications will become apparent to those skilled in the art once the foregoing disclosure is fully understood. The following claims are intended to be interpreted as covering all such variations and modifications.

Claims

1. An apparatus, the apparatus comprising: Multiple computing circuits, each computing circuit including circuitry configured to process work blocks; and The circuit is configured as follows: Receive one or more work blocks for allocation to one or more computing circuits among the plurality of computing circuits; Determine the threshold number of idle computing circuits that are allowed to be activated simultaneously; as well as A certain number of the one or more working blocks, not exceeding the threshold number, are allocated to the idle computing circuit.

2. The apparatus of claim 1, wherein the circuit is further configured to determine the threshold number of idle computing circuits that are allowed to be activated simultaneously based on a comparison of the number of idle computing circuits with the number of the plurality of computing circuits.

3. The apparatus of claim 2, wherein the circuit is further configured to: Receive an indication of the number of computational circuits that have not yet begun executing previously assigned work blocks; and The threshold number of idle computing circuits that are allowed to be activated simultaneously is determined at least in part based on the indication.

4. The apparatus of claim 2, wherein the circuit is further configured to: Receive an indication of the number of computational circuits that have completed the execution of work blocks; and The threshold number of idle computing circuits that are allowed to be activated simultaneously is determined at least in part based on the indication.

5. The apparatus of claim 1, wherein the circuit is further configured to reduce the number of thresholds of idle computing circuits that can be activated simultaneously in response to receiving an indication of a non-zero voltage drop measurement.

6. The apparatus of claim 1, wherein the circuit is further configured to compare the threshold number of idle computing circuits that can be activated simultaneously with the number of idle computing circuits in response to the expiration of a time period since the most recent scheduling window.

7. The apparatus according to claim 1, wherein: Each of the plurality of computing circuits is a Single Instruction Multiple Data (SIMD) circuit that includes multiple execution channels; and Each work block is a wavefront that includes multiple work items.

8. A method, the method comprising: A circuit processing work block composed of multiple computing circuits; The scheduler circuitry receives one or more work blocks to allocate to one or more computing circuits among the plurality of computing circuits; The scheduler determines the threshold number of idle computing circuits that are allowed to be activated simultaneously. as well as The scheduler allocates a certain number of the one or more work blocks, not exceeding the threshold number, to idle computing circuits.

9. The method of claim 8, further comprising determining, by the scheduler, the threshold number of idle computing circuits allowed to be activated simultaneously based on a comparison between the number of idle computing circuits and the number of the plurality of computing circuits.

10. The method according to claim 9, further comprising: The scheduler receives an indication of the number of computational circuits that have not yet begun executing previously assigned work blocks; as well as The scheduler determines the threshold number of idle computing circuits that are allowed to be activated simultaneously, at least in part, based on the indication.

11. The method according to claim 9, further comprising: The scheduler receives an indication of the number of computational circuits that have completed the execution of work blocks; as well as The scheduler determines the threshold number of idle computing circuits that are allowed to be activated simultaneously, at least in part, based on the indication.

12. The method of claim 8, further comprising, in response to receiving an indication of a non-zero voltage drop measurement, the scheduler reducing the number of idle computing circuits that can be activated simultaneously.

13. The method of claim 8, further comprising, in response to the expiration of a time period since the most recent scheduling window, having the scheduler compare the threshold number of idle computing circuits that can be activated simultaneously with the number of idle computing circuits.

14. The method according to claim 8, wherein: Each of the plurality of computing circuits is a Single Instruction Multiple Data (SIMD) circuit that includes multiple execution channels; and Each work block is a wavefront that includes multiple work items.

15. A computing system, the computing system comprising: processor; Multiple chiplets, each chiplet including one or more computing circuits, the one or more computing circuits including circuitry configured to process work blocks; and A scheduler, the scheduler including circuitry, the circuitry being configured to: Receive one or more working blocks to allocate to one or more computing circuits of the plurality of chiplets; Determine the threshold number of idle computing circuits that are allowed to be activated simultaneously; as well as A certain number of the one or more working blocks, not exceeding the threshold number, are allocated to the idle computing circuit.

16. The computing system of claim 15, wherein the scheduler is further configured to determine the threshold number of idle computing circuits that are allowed to be activated simultaneously based on a comparison of the number of idle computing circuits with the number of the plurality of chiplets.

17. The computing system of claim 16, wherein the scheduler is further configured to: Receive an indication of the number of computational circuits that have not yet begun executing previously assigned work blocks; and The threshold number of idle computing circuits that are allowed to be activated simultaneously is determined at least in part based on the indication.

18. The computing system of claim 16, wherein the scheduler is further configured to: Receive an indication of the number of computational circuits that have completed the execution of work blocks; and The threshold number of idle computing circuits that are allowed to be activated simultaneously is determined at least in part based on the indication.

19. The computing system of claim 15, wherein the scheduler is further configured to reduce the number of idle computing circuits that can be activated simultaneously in response to receiving an indication of a non-zero voltage drop measurement.

20. The computing system of claim 15, wherein the scheduler is further configured to compare the threshold number of idle computing circuits that can be activated simultaneously with the number of idle computing circuits in response to the expiration of a time period since the most recent scheduling window.

Images