Real-time GPU rendering with guaranteed power management performance

Abstract

Systems, apparatuses, and methods for performing real-time video rendering with guaranteed power management performance are disclosed. A system includes at least a software driver, a power management unit, and a plurality of processing elements for performing rendering tasks. The system receives input corresponding to rendering tasks that need to be performed. The software driver monitors the received input and the number of rendering tasks to which the input corresponds. The software driver also monitors the amount of time remaining until the next video sync signal. The software driver determines which performance setting will minimize power consumption while still allowing enough time to complete the rendering tasks for the current frame before the next video sync signal. The software driver then causes the management unit to provide the performance setting to the plurality of processing elements while the plurality of processing elements perform rendering tasks for the current frame.

Description

BACKGROUND

[0001] Description of the Related Art

[0002] Various applications rely on real-time rendering of images or video content. For example, cloud gaming, virtual reality, and game viewing behavior are examples of applications that involve real-time rendering of content. Real-time rendering of video frames typically uses a large amount of processing resources that consume a large amount of power. In a real-time rendering environment, the need to control the latency of the generated image frames and the desire to avoid frame drops place special requirements on power management. On one hand, it is desirable to run at as high a clock rate as possible to minimize latency and guarantee that the rendering of images is completed on time. On the other hand, if the processing hardware starts to overheat or approaches a thermal threshold, the hardware will reduce its clock rate, which then leads to frame drops. These issues are particularly challenging for power or thermal constrained platforms.

[0003] Various frame-based real-time applications, including gaming applications and other types of rendering applications, submit multiple jobs per frame and repeat the process at a constant or variable frame rate. Depending on the application runtime behavior, the complexity and computational requirements of the per-frame processing unit workload (e.g., number of jobs, time per job, resources per job) can vary. For such applications, either the processing unit completes frame execution early enough to allow the frame to be used in time (e.g., displayed or transmitted), or the processing unit completes frame execution later, which results in the frame being dropped or consumed later. Such delays can negatively impact the user experience.

[0004] In view of the above, there is a need for improved methods for managing real-time video rendering with guaranteed power management performance. BRIEF DESCRIPTION OF DRAWINGS

[0005] The advantages of the methods and mechanisms described herein can be better understood by reference to the following description in conjunction with the following drawings, in which:

[0006] Figure 1 is a block diagram of one implementation of a computing system.

[0007] Figure 2 is a block diagram of one implementation of a computing system.

[0008] Figure 3 is a timing diagram of one implementation of selecting a performance setting for a frame being rendered based on queue occupancy.

[0009] Figure 4 is an example of a table for mapping the number of incoming tasks and the remaining time to a performance setting according to one implementation.

[0010] Figure 5is a generalized flow diagram illustrating one implementation of a method for performing real-time video rendering with guaranteed power management performance.

[0011] Figure 6 is a generalized flow diagram illustrating one implementation of a method for controlling performance settings of processing hardware based on application type. DETAILED DESCRIPTION

[0012] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, it will be apparent to those skilled in the art that the various implementations can be practiced without such specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail, so as not to obscure the described methods. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements can be exaggerated relative to other elements.

[0013] Systems, devices, and methods are disclosed herein for use case driven event-based feedforward control windows to enable real-time GPU rendering with guaranteed power management governor performance. In one implementation, a system includes at least a software driver, a power management unit, and one or more processing elements for performing rendering tasks. The system receives input corresponding to rendering tasks that need to be performed. The software driver monitors the number of inputs received and the number of rendering tasks to which the inputs correspond. The software driver also monitors the amount of time remaining until the next video synchronization signal. The software driver determines which performance setting will minimize power consumption while still allowing enough time to complete the rendering tasks for the current frame before the next video synchronization signal. The software driver then causes the power management unit to provide the performance setting to the plurality of processing elements while the plurality of processing elements perform the rendering tasks for the current video frame.

[0014] Reference is now made to Figure 1 FIG. 1 shows a block diagram of one implementation of a computing system 100. In one implementation, the computing system 100 includes at least processors 105A-105N, a control unit 110, input / output (I / O) interfaces 120, a bus 125, one or more memory controllers 130, a network interface 135, one or more memory devices 140, a power supply 145, a power management unit 150, a display controller 160, and a display 165. In other implementations, the computing system 100 includes other components, and / or the computing system 100 is arranged differently. The processors 105A-105N represent any number of processors included in the system 100, the number of processors varying from implementation to implementation.

[0015] In one implementation, processor 105A is a general-purpose processor, such as a central processing unit (CPU). In one implementation, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and the like. In one implementation, processor 105N is a GPU that provides pixels to display controller 160 for driving to display 165. In some implementations, processors 105A-105N include multiple data parallel processors. In one implementation, control unit 110 is a software driver executing on processor 105A. In other implementations, control unit 110 includes control logic independent of and / or incorporated within processors 105A-105N. In general, control unit 110 is any suitable combination of software and / or hardware.

[0016] One or more memory controllers 130 represent any number and type of memory controllers accessible by processors 105A-105N. One or more memory controllers 130 are coupled to any number and type of memory devices 140. One or more memory devices 140 represent any number and type of memory devices. For example, the types of memory in one or more memory devices 140 include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), and the like.

[0017] I / O interface 120 represents any number and type of I / O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I / O interface 120. Such peripheral devices include, but are not limited to, displays, keyboards, mice, printers, scanners, media recording devices, external storage devices, network interface cards, and the like. Network interface 135 is used to receive and send network messages across a network. Bus 125 represents any type of bus or fabric with any number of links for connecting the different components of system 100 together.

[0018] In one implementation, the one or more queues 142 are stored in the one or more memory devices 140. In other implementations, the one or more queues 142 are stored in other locations within the system 100. The one or more queues 142 represent any number and type of queues allocated in the system 100. In one implementation, the one or more queues 142 store rendering tasks to be performed for a frame that is being rendered. In one implementation, the rendering tasks are queued in the one or more queues 142 based on input received via the network interface 135. For example, in one scenario, a user of a video game application generates input and sends it through a network (not shown) to the system 100. In another implementation, the input is generated by a peripheral device connected to the I / O interface 120.

[0019] In one implementation, the power management unit 150 supplies power from the power supply 145 to the components of the system 100, and the power management unit 150 controls the various power performance states of the components within the system 100. In response to receiving the update from the control unit 110, the power management unit 150 causes other components within the system 100 to increase or decrease their current power performance state. In various implementations, changing the power performance state includes changing the current operating frequency of the device and / or changing the current voltage level of the device. When the power performance state of the processors 105A-105N is decreased, this causes the computing tasks performed by the processors 105A-105N to take longer to complete.

[0020] In one implementation, the control unit 110 sends a command to the power management unit 150 to cause the processor 105N to operate at a relatively higher power performance state in response to determining that the number of rendering tasks for the current frame is greater than a given threshold. In one implementation, the given threshold is adjusted based on the amount of time remaining until the next video synchronization signal. For example, the shorter the time remaining until the next video synchronization signal, the lower the given threshold is programmed to be.

[0021] In various implementations, the computing system 100 is any one of a computer, a laptop, a mobile device, a server, or various other types of computing systems or devices. It should be noted that the number of components of the computing system 100 varies from implementation to implementation. For example, in other implementations, there are more or fewer numbers of each component than shown in Figure 1 It should also be noted that, in other implementations, the computing system 100 includes other components not shown in Figure 1 and / or one or more of the components shown in the computing system 100 are omitted. Additionally, in other implementations, the computing system 100 is structured in a different manner than shown in Figure 1

[0022] Turning now to​Figure 2 FIG. 3 illustrates a block diagram of another implementation of a computing system 200. In one implementation, the system 200 includes a GPU 205, a system memory 225, and a local memory 230 belonging to the GPU 205. The system 200 also includes other components, which are not shown to avoid obscuring the drawing. The GPU 205 includes at least a command processor 235, a scheduler 250, compute units 255A-255N, a memory controller 220, a global data share 270, a level one (LI) cache 265, and a level two (L2) cache 260. Note that the compute units 255A-255N can also be referred to herein as “processing elements.” In other implementations, the GPU 205 includes other components, omits one or more of the illustrated components, has multiple instances of a component, even if only one instance is shown in the figures, and / or is organized in other suitable ways. Figure 2 In one implementation, the circuitry of the GPU 205 is included in the processor 105N (of FIG. 1). Figure 1 In one implementation, the circuitry of the GPU 205 is included in the processor 105N (of FIG. 1).

[0023] In various implementations, the computing system 200 executes any of various types of software applications. As part of executing a given software application, a host CPU (not shown) of the computing system 200 launches a rendering task to be performed on the GPU 205. The command processor 235 receives commands from the host CPU and issues corresponding rendering tasks to the compute units 255A-255N using the scheduler 250. The rendering tasks executing on the compute units 255A-255N read and write data to the global data share 270, the LI cache 265, and the L2 cache 260 within the GPU 205. Although not shown in FIG. 3, in one implementation, the compute units 255A-255N also include one or more caches and / or local memories within each compute unit 255A-255N. Figure 2 In various implementations, the compute units 255A-255N execute any number of frame-based applications that are rendering frames to be displayed, streamed, or consumed in real time. In one implementation, one or more queues 232 are stored in the local memory 230. In other implementations, the one or more queues 232 are stored in other locations within the system 200. The one or more queues 232 represent any number and type of queues allocated in the system 200. In one implementation, the one or more queues 232 store rendering tasks to be performed by the GPU 205.

[0024] In one implementation, the performance setting of GPU 205 is adjusted based on the number of rendering tasks for the current frame stored in one or more queues 232 and based on the amount of time remaining until the next video sync signal. In various implementations, the performance setting of GPU 205 is adjusted so that the rendering tasks are completed by the next video sync signal while also achieving a power consumption goal. In one implementation, the performance setting is adjusted by a control unit (not shown). The control unit can be a software driver executing on a CPU (not shown) or the control unit can include control logic implemented within a programmable logic device (e.g., an FPGA) or as dedicated hardware (e.g., an ASIC). In some cases, the control unit includes a combination of software and hardware.

[0025] In one implementation, the performance setting of GPU 205 corresponds to a particular power setting, power state, or operating point of GPU 205. In one implementation, the control unit uses dynamic voltage and frequency scaling (DVFS) to change the frequency and / or voltage of GPU 205 to limit power consumption to a selected power allocation. Each individual frequency and voltage setting can correspond to a separate performance setting. In one implementation, the performance setting selected by the control unit controls a phase-locked loop (PLL) unit (not shown) that generates and distributes a corresponding clock signal to GPU 205. In one implementation, the performance setting selected by the control unit controls a voltage regulator (not shown) that provides a supply voltage to GPU 205. In other implementations, other mechanisms can be used to change the operating point and / or power setting of GPU 205 in response to receiving a command from the control unit to reach a particular performance setting.

[0026] Referring now to Figure 3 , a timing diagram of one implementation of selecting a performance setting for a frame being rendered based on queue occupancy is shown. When rendering frames of a video sequence, in one implementation, a software driver changes the performance setting of the rendering hardware based at least in part on queue occupancy. Queue occupancy refers to the number of rendering tasks that have been queued for processing hardware (e.g., a GPU) for the current frame being rendered.

[0027] A frame period is shown, which is defined by video sync signals (or VSync) corresponding to the start and end of each frame being rendered. In Figure 3The initial performance setting 325 can be a default setting in one implementation. In another implementation, the initial performance setting 325 can be programmed based on the type of application, a hint generated by the application, an estimate of the complexity of the current frame being rendered, and / or based on other factors. In one implementation, a software driver responsible for controlling the performance setting monitors the queue occupancy of one or more rendering task queues. The software driver monitors the queue occupancy multiple times within each frame period, and the frequency of the monitoring can be fixed or programmable depending on the implementation. As Figure 3 As shown, the first occupancy sample 305 specifies a particular number of rendering tasks that have been queued. Based on this sample 305, the software driver maintains the current performance setting 325.

[0028] The next queue occupancy sample 310 is less than the previous sample 305. This indicates that the number of rendering tasks has decreased as the processing hardware has completed one or more rendering tasks. Accordingly, in response to detecting a decrease in queue occupancy from sample 310 to sample 305, the software driver decreases the performance setting 330 to reduce the power consumption of the processing hardware. In the next two samples 315 and 320, this trend continues with the software driver decreasing the power for performance settings 335 and 340, respectively. This decrease in performance setting is acceptable because there are fewer rendering tasks for the processing hardware to complete for the current frame. When the video synchronization signal occurs, the current frame is sent to the display, over the network to one or more clients, or to other locations.

[0029] For the next frame period, the first occupancy sample 345 indicates that the rendering tasks for this frame are relatively few. Accordingly, the performance setting 340 for the processing hardware can remain at a relatively low level at the beginning of the frame period. The next occupancy sample 350 indicates that the number of rendering tasks has decreased, allowing for a lower performance setting 370. However, the subsequent occupancy sample 355 indicates that the queue occupancy has increased. This can be due to the receipt of multiple rendering tasks, which can be caused by player input in a game scenario, user movement in a virtual reality environment, or other input or events generated in other types of applications.

[0030] When the software driver detects an increase in the occupancy of occupancy sample 355 and a decrease in the remaining time until the next video sync signal, the software driver responds by increasing the power provided to the processing hardware to a performance setting 375. In one implementation, performance setting 375 is a maximum performance setting for the processing hardware. The next two occupancy samples 360 and 365 indicate that the number of rendering tasks for the current frame has decreased. However, the time available to complete these rendering tasks has also decreased, which means that the software driver will maintain a relatively high performance setting 375 for the processing hardware.

[0031] The example shown in timing diagram 300 indicates one particular implementation in which the software driver adjusts a performance setting based on the queue occupancy of one or more rendering task queues. In other implementations, the software driver can make other types of adjustments based on changes in queue occupancy. It should be understood that the granularity of performance setting updates can vary depending on the implementation. Furthermore, the frequency with which the software driver checks queue occupancy can also vary depending on the implementation.

[0032] Turning now to FIG. 4, Figure 4 shows one implementation of a table 400 for mapping the number of incoming tasks and the remaining time to a performance setting. In one implementation, control logic or a software driver performs a lookup of columns 405 and 410 of table 400 to retrieve a corresponding performance setting. The retrieved performance setting is used to program a plurality of processing elements (e.g., GPU 205 of FIG. 2) to operate at a particular operating point. In one implementation, column 405 includes different possible values for the number of incoming tasks (e.g., rendering tasks). In other implementations, column 405 includes other values that represent the amount of work that needs to be performed to render a current frame. For example, in another implementation, column 405 is measured in terms of queue occupancy. In other implementations, column 405 is measured in terms of the number of hints received, the number of events detected, or otherwise. In one implementation, column 410 includes entries for different amounts of remaining time until the next video sync signal. Figure 2

[0033] In one implementation, the software driver uses the number of rendering tasks and the amount of remaining time until the next video sync signal to perform a lookup of table 400. If the lookup results in a hit, the performance setting is retrieved from the matching entry. If the lookup results in a miss, the software driver can interpolate a performance setting value based on the two most recent entries. After retrieving and / or calculating a particular performance setting, the software driver causes the rendering hardware to operate at the particular performance setting. In one implementation, the software driver performs multiple lookups of table 400 per frame to update the performance setting as the number of rendering tasks and / or the amount of remaining time changes during a frame period.​

[0034] In one implementation, there is a separate table 400 for each different application that can run on the system. For example, for a cloud gaming environment, the system stores a first table 400A. For a virtual reality application, the system stores a second table 400B. The system can also store any number of other tables 400C through 400N for different applications. Each application can have different characteristics and complexity for the rendering tasks that can be performed while the application is executing. Accordingly, each application has a separate table 400 to accommodate the different performance settings that should be used based on the number of rendering tasks and the time remaining.

[0035] In one implementation, each table 400 is programmed by software. The table 400 can be programmed based on test data, and / or the table 400 can be programmed based on real-time training based on monitoring application behavior. For example, in one implementation, the table 400 is programmed by software with default values for a given application. Then, during runtime, the software can monitor the given application to see if any changes have been observed in the runtime environment compared to the test scenarios used to generate the default values for the table 400. If the rendering tasks are taking longer than predicted, or if the rendering tasks are completing earlier than predicted, the values stored in the performance setting column 415 can be updated to more accurately reflect the behavior of the given application. In another implementation, instead of using a table 400 to select a performance setting, the software driver uses a formula to calculate a performance setting based on the number of incoming tasks and the time remaining. In other implementations, the software driver uses other suitable techniques to select a performance setting.

[0036] Turning now to Figure 5 , one implementation of a method 500 for performing real-time video rendering with guaranteed power management performance is shown. For purposes of discussion, the steps in this implementation and those of Figure 6 are shown in sequential order. However, it is to be appreciated that one or more of the elements can be performed concurrently, in a different order than shown, or omitted entirely, in various implementations of the method. Other additional elements can also be performed as desired. Any of the various systems or devices described herein are configured to implement the method 500.

[0037] The software driver monitors input corresponding to the rendering task for the current frame being rendered (block 505). In one implementation, the input is an event associated with a user on a network. For example, the user is playing a video game in a cloud gaming scenario. In another implementation, the input is user movement in a virtual reality environment. In other implementations, other types of input for other types of scenarios are received in block 505. Further, the software driver monitors an amount of time remaining in the current frame period before the next video sync signal (block 510). Next, the software driver determines a lowest possible performance setting for completing the incoming rendering task in the amount of time remaining before the next video sync signal (block 515). In one implementation, the performance settings of the processing hardware (e.g., GPU) include corresponding voltage and frequency values.

[0038] If the software driver determines that the incoming rendering task cannot be completed in the amount of time remaining before the next video sync signal even at the maximum performance setting (conditional block 520, "No" branch), the software driver causes the previous frame to be replayed and the processing hardware to be set to an idle or relatively low performance setting (e.g., the lowest possible performance setting) (block 525). Alternatively, the software driver can cause the current frame to be delayed in block 525 instead of replaying the previous frame. If there is a performance setting that allows the incoming rendering task to be completed in the amount of time remaining before the next Vsync (conditional block 520, "Yes" branch), the software driver causes the processing hardware to operate at the given performance setting (block 530). In one implementation, the given performance setting is the lowest possible performance setting for completing the incoming rendering task in the amount of time remaining before the next video sync signal. In another implementation, the given performance setting is one setting above the lowest possible performance setting to provide a margin of error for completing the incoming rendering task in the amount of time remaining before the next video sync signal. In other implementations, the margin of error can increase to two or more settings above the lowest possible performance setting.

[0039] If the rendering of the current frame has not completed (conditional block 540, "No" branch), the method 500 returns to block 505 after a certain amount of time has elapsed or after a certain event is detected (i.e., a change in queue occupancy). It should be noted that some hysteresis can be added to the loop to prevent the performance setting from being changed too frequently. If the rendering of the current frame has completed (conditional block 540, "Yes" branch), the method 500 ends. It should be noted that the method 500 can be performed for each video frame of a video sequence being rendered.

[0040] Turning now to Figure 6This illustrates one implementation of a method 600 for controlling performance settings of processing hardware based on application type. The control unit determines which application the system is currently executing (box 605). The control unit then loads a performance settings lookup table corresponding to the application (e.g., ...). Figure 4 The table is referred to in box 610 (400). Next, the control unit uses the table to select performance settings for the processing hardware based on queue occupancy and the amount of time remaining before the next video synchronization signal (box 615). If the control unit detects that the system is executing a different application (condition box 620, "Yes" branch), method 600 returns to box 610. Otherwise, if the system continues to execute the given application (condition box 620, "No" branch), method 600 returns to box 615. It should be noted that in some cases, a single application may have multiple different performance setting lookup tables. For example, a video game application may have different scenes with varying rendering complexities. For the first scene of the application, the control unit may load the first table, for the second scene, the control unit loads the second table, and so on.

[0041] In various implementations, program instructions from a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose processor or a special-purpose processor are envisioned. In various implementations, such program instructions may be represented by a high-level programming language. In other implementations, the program instructions may be compiled from a high-level programming language into binary, intermediate, or other forms. Alternatively, program instructions describing the behavior or design of the hardware may be written. Such program instructions may be represented by a high-level programming language such as C. Alternatively, a hardware design language (HDL), such as Verilog, may be used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. The storage media may be accessed by the computing system during use to provide the program instructions to the computing system for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0042] It should be emphasized that the above implementations are merely non-limiting examples. Many variations and modifications will be apparent to those skilled in the art once the above disclosure is fully understood. The appended claims are intended to cover all such variations and modifications.

Claims

1. A power management unit configured to determine a required performance level, comprising: a control unit configured to: monitor, for a current video frame, a number of rendering tasks to be performed; determine an amount of time remaining until a next synchronization event associated with the frame; access data identifying a performance setting associated with a combination of the number of rendering tasks and the amount of time remaining until a next video synchronization event; retrieve the performance setting corresponding to the number of rendering tasks and the amount of time remaining; cause processing hardware to perform the rendering tasks while operating at the retrieved performance setting.

2. The power management unit of claim 1, wherein the control unit is configured to cause a previous frame to be replayed in response to determining that the rendering tasks cannot be completed before the next synchronization signal.

3. The power management unit of claim 1, wherein the control unit is configured to cause processing hardware to operate at a lowest available performance setting in response to determining that the rendering tasks can be completed before the next synchronization signal while operating at the lowest available performance setting.

4. The power management unit of claim 1, wherein the data is stored in a table comprising a plurality of entries, wherein each entry of the table represents an indexable combination of a number of tasks, an amount of time remaining, and a corresponding performance setting.

5. The power management unit of claim 4, wherein, a given entry in the table includes a performance setting corresponding to the number of rendering tasks and the amount of time remaining until a next synchronization event.

6. The power management unit of claim 5, wherein, the control unit is configured to update values stored in entries of the table based on application behavior.

7. The power management unit of claim 4, further comprising a plurality of tables, wherein each table corresponds to a different application and one or more of scenarios for a given application.

8. A method for determining a required performance level, the method comprising: monitoring, for a current video frame, a number of rendering tasks to be performed; determining an amount of time remaining until a next synchronization event associated with the frame; accessing data identifying a performance setting associated with a combination of the number of rendering tasks and the amount of time remaining until a next video synchronization event; retrieving the performance setting corresponding to the number of rendering tasks and the amount of time remaining; causing processing hardware to perform the rendering tasks while operating at the retrieved performance setting.

9. The method of claim 8, comprising causing a previous frame to be replayed in response to determining that the rendering tasks cannot be completed before the next synchronization signal.

10. The method of claim 8, further comprising causing processing hardware to operate at a lowest available performance setting in response to determining that the rendering tasks can be completed before the next synchronization signal while operating at the lowest available performance setting.

11. The method of claim 8, wherein the data is stored in a table comprising a plurality of entries.

12. The method of claim 11, wherein each entry of the table represents an indexable combination of a number of tasks, an amount of time remaining, and a corresponding performance setting.

13. The method of claim 12, comprising updating values stored in entries of the table based on application behavior.

14. The method of claim 11, further comprising a plurality of tables, wherein each table corresponds to a different application program and one or more of the scenarios for a given application program.

Images