Real-time GPU rendering with performance-guaranteed power management

Abstract

A system, apparatus, and method for performing real-time video rendering with performance-guaranteed power management are disclosed. The system includes at least a software driver, a power management unit, and multiple processing elements that perform rendering tasks. The system receives inputs corresponding to rendering tasks that need to be performed. The software driver monitors the received inputs and the number of rendering tasks they correspond to. The software driver also monitors the remaining time until the next video synchronization signal. The software driver determines a performance setting that minimizes power consumption while still ensuring sufficient time to finish the rendering tasks for the current frame before the next video synchronization signal. The software driver then causes the power management unit to provide this performance setting to the multiple processing elements as they perform the rendering tasks for the current frame.

Description

[Background technology]

[0001] Description of Related Art Various applications rely on real-time rendering of image or video content. For example, cloud gaming, virtual reality, and game viewing are examples of applications that require real-time rendering of content. The real-time rendering of video frames uses significant processing resources, which often consumes a lot of power. In a real-time rendering environment, the requirement to control the latency of generated image frames and the desire to avoid dropping frames impose special demands on power management. On the one hand, it is desirable to run at the highest possible clock rate to minimize latency and ensure that image rendering is completed on time. On the other hand, if the processing hardware begins to overheat or approaches a temperature threshold, the hardware will reduce the clock rate, resulting in dropped frames. These issues are particularly challenging on power- or thermally-constrained platforms.

[0002] Various frame-based real-time applications include not only gaming applications but also other types of rendering applications that submit multiple jobs for each frame and repeat this processing at a constant or variable frame rate. The processing unit's workload per frame (e.g., number of jobs, time per job, resources per job) can vary in complexity and computational demand depending on the runtime behavior of the application. In such applications, the processing unit either completes execution of a frame early enough so that the frame can be used (e.g., displayed or transmitted) in a timely manner, or the processing unit delays in completing execution of a frame, resulting in the frame being dropped or being consumed late. Such delays adversely affect the user experience.

[0003] In view of the above, an improved method for managing real-time video rendering with performance-guaranteed power management is desired.

[0004] The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computing system. [Figure 2] FIG. 1 is a block diagram of one embodiment of a computing system. [Figure 3] FIG. 10 is a timing diagram of one embodiment of selecting a performance setting for a rendered frame based on queue occupancy. [Figure 4] FIG. 10 illustrates an example of a table mapping input task counts and remaining time to performance settings, according to one embodiment. [Figure 5] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for real-time video rendering with performance-guaranteed power management. [Figure 6] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for controlling performance settings of processing hardware based on application type. DETAILED DESCRIPTION OF THE INVENTION

[0006] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It should be understood that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements.

[0007] Disclosed herein are systems, apparatus, and methods for implementing real-time GPU rendering using a performance-guaranteed power management governor based on a use-case-driven event-based feed-forward control window. In one embodiment, the system includes at least a software driver, a power management unit, and one or more processing elements for executing rendering tasks. The system receives inputs corresponding to rendering tasks that need to be executed. The software driver monitors the number of received inputs and the number of rendering tasks they correspond to. The software driver also monitors the remaining time until the next video synchronization signal. The software driver determines a performance setting that minimizes power consumption while ensuring sufficient time to finish the rendering task for the current frame before the next video synchronization signal. The software driver then causes the power management unit to provide this performance setting when the processing elements execute the rendering task for the current video frame.

[0008] 1, a block diagram of one embodiment of computing system 100 is shown. In one embodiment, computing system 100 includes at least processors 105A-105N, control unit 110, input / output (I / O) interface 120, bus 125, memory controller(s) 130, network interface 135, memory device(s) 140, power supply 145, power management unit 150, display controller 160, and display 165. In other embodiments, computing system 100 includes other components and / or is configured differently. Processors 105A-105N represent any number of processors that may be included in system 100, with the number of processors varying depending on the embodiment.

[0009] In one embodiment, processor 105A is a general-purpose processor such as a central processing unit (CPU). In one embodiment, processor 105N is a data-parallel processor with a highly parallel architecture. Data-parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc. In one embodiment, processor 105N is a GPU that provides pixels to display controller 160 for transmission to display 165. In some embodiments, processors 105A-105N include multiple data-parallel processors. In one embodiment, control unit 110 is a software driver running on processor 105A. In other embodiments, control unit 110 includes control logic separate from processors 105A-105N and / or embedded within processors 105A-105N. Generally speaking, control unit 110 is any suitable combination of software and / or hardware.

[0010] Memory controller(s) 130 represent any number and type of memory controller accessible by processors 105A-105N. Memory controller(s) 130 are coupled to any number and type of memory device(s) 140. Memory device(s) 140 represent any number and type of memory devices. For example, the type of memory in memory device(s) 140 may include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), or other memory.

[0011] I / O interface 120 represents any number and type of I / O interface (e.g., Peripheral Component Interconnect (PCI) bus, PCI Extended (PCI-X), PCI Express (PCIE) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices (not shown) are coupled to I / O interface 120. Such peripheral devices include, but are not limited to, displays, keyboards, mice, printers, scanners, media recording devices, external storage devices, network interface cards, etc. Network interface 135 is used to send and receive network messages over a network. Bus 125 represents any type of bus or fabric having any number of links for connecting different components of system 100 to one another.

[0012] In one embodiment, queue(s) 142 are stored in memory device(s) 140. In other embodiments, queue(s) 142 are stored elsewhere in system 100. Queue(s) 142 represent any number and type of queue(s) allocated within system 100. In one embodiment, queue(s) 142 store rendering tasks to be performed for frames to be rendered. In one embodiment, rendering tasks are enqueued into queue(s) 142 based on input received via network interface 135. For example, in one scenario, input is generated by a user of a video game application and sent to system 100 over a network (not shown). In another embodiment, input is generated by a peripheral device connected to I / O interface 120.

[0013] In one embodiment, power management unit 150 provides power to the components of system 100 from power supply 145, and power management unit 150 controls various power and performance states of the components within system 100. In response to receiving update data from control unit 110, power management unit 150 causes other components within system 100 to either increase or decrease their current power and performance states. In various embodiments, changing the power and performance state includes changing the current operating frequency of a device and / or changing the current voltage level of a device. As the power and performance state of processors 105A-105N decreases, computing tasks being performed by processors 105A-105N take longer to complete.

[0014] In one embodiment, in response to determining that the number of rendering tasks for the current frame is greater than a predetermined threshold, the control unit 110 commands the power management unit 150 to operate the processor 105N in a relatively high power performance state. In one embodiment, the predetermined threshold is adjusted based on the remaining time until the next video synchronization signal. For example, the predetermined threshold is programmed to be lower the shorter the time until the next video synchronization signal.

[0015] In various embodiments, computing system 100 is a computer, a laptop, a mobile device, a server, or various other types of computing systems or devices. It should be noted that the number of components in computing system 100 may vary depending on the embodiment. For example, in other embodiments, the number of each component may be greater or less than the number shown in FIG. 1 . It should be noted that in other embodiments, computing system 100 may include other components not shown in FIG. 1 and / or may omit one or more of the components shown in computing system 100. Furthermore, in other embodiments, computing system 100 may be configured differently than that shown in FIG. 1 .

[0016] Referring to FIG. 2, a block diagram of another embodiment of a computing system 200 is shown. In one embodiment, the system 200 includes a GPU 205, a system memory 225, and a local memory 230 belonging to the GPU 205. The system 200 also includes other components not shown to avoid obscuring the diagram. The GPU 205 includes at least a command processor 235, a scheduler 250, compute units 255A-255N, a memory controller 220, a global data share 270, a level 1 (L1) cache 265, and a level 2 (L2) cache 260. Note that the compute units 255A-255N may be referred to herein as "multiple processing elements." In other embodiments, the GPU 205 includes other components, omits one or more of the components shown, has multiple instances of components even when only one instance is shown in FIG. 2, and / or is organized in any other suitable manner. In one embodiment, the circuitry of GPU 205 is included in processor 105N (of FIG. 1).

[0017] In various embodiments, computing system 200 executes any of various types of software applications. As part of executing a given software application, a host CPU (not shown) of computing system 200 initiates rendering tasks to be executed on GPU 205. Command processor 235 receives commands from the host CPU and, using scheduler 250, issues corresponding rendering tasks to compute units 255A-255N. Rendering tasks executing on compute units 255A-255N read and write data from a global data share 270, an L1 cache 265, and an L2 cache 260 within GPU 205. While not shown in FIG. 2 , in one embodiment, compute units 255A-255N include one or more caches and / or local memories within each compute unit 255A-255N. In various embodiments, compute units 255A-255N execute any number of frame-based applications that render frames to be displayed, streamed, or consumed in real time. In one embodiment, queue(s) 232 are stored in local memory 230. In other embodiments, queue(s) 232 are stored elsewhere in system 200. Queue(s) 232 represent any number and type of queues allocated within system 200. In one embodiment, queue(s) 232 store rendering tasks to be performed by GPU 205.

[0018] In one embodiment, the performance settings of the GPU 205 are adjusted based on the number of rendering tasks for the current frame stored in queue(s) 232 and based on the time remaining until the next video sync signal. In various embodiments, the performance settings of the GPU 205 are adjusted to finish rendering tasks before the next video sync signal while achieving power consumption targets. In one embodiment, the performance settings are adjusted by a control unit (not shown). The control unit may be a software driver running on a CPU (not shown), or the control unit may include control logic implemented in a programmable logic device (e.g., FPGA) or as dedicated hardware (e.g., ASIC). In some cases, the control unit includes a combination of software and hardware.

[0019] In one embodiment, the performance setting of the GPU 205 corresponds to a particular power setting, power state, or operating point of the GPU 205. In one embodiment, the control unit uses dynamic voltage and frequency scaling (DVFS) to change the frequency and / or voltage of the GPU 205 to limit power consumption to a selected power budget. Each frequency and voltage setting may correspond to a separate performance setting. In one embodiment, the performance setting selected by the control unit controls a phase-locked loop (PLL) unit (not shown) that generates and distributes a corresponding clock signal to the GPU 205. In one embodiment, the performance setting selected by the control unit controls a voltage regulator (not shown) that provides a supply voltage to the GPU 205. In other embodiments, other mechanisms may be used to change the operating point and / or power setting of the GPU 205 in response to receiving a command from the control unit to reach a particular performance setting.

[0020] 3, a timing diagram of one embodiment for selecting performance settings for a rendered frame based on queue occupancy is shown. When rendering a frame of a video sequence, in one embodiment, a software driver changes performance settings of the rendering hardware based at least in part on the queue occupancy. Queue occupancy refers to the number of rendering tasks enqueued in the processing hardware (e.g., a GPU) for the current frame being rendered.

[0021] A frame period is shown, bounded by video synchronization signals (or VSync) corresponding to the start and end of each frame being rendered. In the first frame period shown in FIG. 3 , initial performance settings 325 for the frame being rendered are set in the processing hardware. The initial performance settings 325 may be default settings in one embodiment. In another embodiment, the initial performance settings 325 are programmable based on the type of application, hints generated by the application, an estimate of the complexity of the current frame being rendered, and / or other factors. In one embodiment, a software driver responsible for controlling the performance settings monitors the queue occupancy of the rendering task queue(s). The software driver monitors the queue occupancy multiple times per frame period. The frequency of the monitoring may be either fixed or programmable, depending on the embodiment. As shown in FIG. 3 , an initial occupancy sample 305 indicates that a certain number of rendering tasks have been enqueued. Based on this sample 305, the software driver maintains the current performance settings 325.

[0022] The next queue occupancy sample 310 is decreased from the previous sample 305, indicating that the number of rendering tasks has decreased due to one or more rendering tasks being completed by the processing hardware. Therefore, in response to detecting the decrease in queue occupancy from sample 310 to sample 305, the software driver reduces performance setting 330 to reduce the power consumption of the processing hardware. This trend continues for the next two samples 315 and 320, where the software driver reduces the power of each of performance settings 335 and 340. This reduction in performance setting is acceptable because the processing hardware has fewer rendering tasks to complete during the current frame. When a video sync signal occurs, the current frame is sent to the display, sent over the network to one or more clients, or sent elsewhere.

[0023] In the next frame period, the first occupancy sample 345 indicates that there are relatively few rendering tasks for this frame. Therefore, performance setting 340 can remain at a relatively low level for the processing hardware at the start of the frame period. The next occupancy sample 350 indicates that the number of rendering tasks is decreasing, allowing for a lower performance setting 370. However, the subsequent occupancy sample 355 indicates that the queue occupancy has increased. This may be due to the receipt of multiple rendering tasks, which may be caused by player input in a gaming scenario, user movement in a virtual reality environment, or other input or events generated in other types of applications.

[0024] When the software driver detects the increased occupancy in occupancy sample 355 and the decreased time remaining until the next video sync signal, it responds by increasing the power supplied to the processing hardware to performance setting 375. In one embodiment, performance setting 375 is the maximum performance setting of the processing hardware. The next two occupancy samples 360 and 365 indicate that the number of rendering tasks for the current frame has decreased. However, the time to complete these rendering tasks has also decreased, meaning that the software driver will maintain the relatively high performance setting 375 of the processing hardware.

[0025] The example shown in timing diagram 300 illustrates one particular embodiment of a software driver adjusting performance settings based on queue occupancy of rendering task queue(s). In other embodiments, the software driver may make other types of adjustments based on changes in queue occupancy. It should be understood that the granularity at which performance setting updates occur may vary from embodiment to embodiment. Also, the frequency at which the software driver checks queue occupancy may vary from embodiment to embodiment.

[0026] Referring to FIG. 4, one embodiment of a table 400 for mapping the number of input tasks and remaining time to performance settings is shown. In one embodiment, control logic or a software driver performs a lookup of columns 405 and 410 of table 400 to obtain the corresponding performance setting. The obtained performance setting is used to program multiple processing elements (e.g., GPU 205 of FIG. 2) to operate at a particular operating point. In one embodiment, column 405 includes different possible values ​​for the number of input tasks (e.g., rendering tasks). In other embodiments, column 405 includes other values ​​that represent the amount of work that needs to be performed to render the current frame. For example, in another embodiment, column 405 is measured in terms of queue occupancy. In other embodiments, column 405 is measured in terms of the number of received hints, the number of detected events, or other aspects. In one embodiment, column 410 includes entries for different remaining times until the next video sync signal.

[0027] In one embodiment, the software driver performs a lookup in table 400 using the number of rendering tasks and the time remaining until the next video sync signal. If the lookup results in a hit, the performance setting is obtained from the matching entry. If the lookup fails, the software driver can interpolate a performance setting value based on the two closest entries. After obtaining and / or calculating a particular performance setting, the software driver operates the rendering hardware at the particular performance setting. In one embodiment, the software driver performs multiple lookups to table 400 per frame to update the performance setting as the number of rendering tasks and / or the time remaining changes during the frame.

[0028] In one embodiment, a separate table 400 exists for each different application that may run on the system. For example, in a cloud gaming environment, a first table 400A is stored by the system. In a virtual reality application, a second table 400B is stored by the system. Any number of other tables 400C-400N may similarly be stored by the system for various applications. Each application may have different characteristics and complexities for the rendering tasks that are likely to be performed when the application is run. To that end, each application has a separate table 400 to accommodate different performance settings to be used based on the number of rendering tasks and remaining time.

[0029] In one embodiment, each table 400 is programmed by software. Table 400 can be programmed based on test data and / or table 400 can be programmed based on real-time training based on monitoring the behavior of the application. For example, in one embodiment, table 400 is programmed by software with default values ​​for a given application. Then, during runtime, the software can monitor the given application to see if any changes in the runtime environment are observed compared to the test scenario used to generate the default values ​​of table 400. If a rendering task takes longer than expected or if a rendering task finishes sooner than expected, the value stored in performance setting column 415 can be updated to more accurately reflect the behavior of the given application. In another embodiment, rather than using table 400 to select a performance setting, the software driver uses a formula to calculate the performance setting based on the number of input tasks and the time remaining. In other embodiments, the software driver uses other suitable techniques to select a performance setting.

[0030] Referring to Figure 5, one embodiment of a method 500 for real-time video rendering with performance-guaranteed power management is shown. For purposes of illustration, the steps of this embodiment and those of Figure 6 are shown sequentially. However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or devices described herein may be configured to perform method 500.

[0031] The software driver monitors input corresponding to the rendering task for the current frame being rendered (block 505). In one embodiment, the input is an event related to a user on the network. For example, a user is playing a video game in a cloud gaming scenario. In another embodiment, the input is a user's movement in a virtual reality environment. In other embodiments, other types of input for other types of scenarios are received at block 505. The software driver also monitors the time remaining in the current frame period until the next video sync signal (block 510). The software driver then determines the lowest possible performance setting to complete the input rendering task in the time remaining until the next video sync signal (block 515). In one embodiment, the performance setting of the processing hardware (e.g., GPU) includes corresponding voltage and frequency values.

[0032] If the software driver determines that the input rendering task cannot be completed at the highest performance setting in the time remaining until the next video sync signal (conditional block 520: “No”), the software driver plays the previous frame and sets the processing hardware to idle or a relatively low performance setting (e.g., the lowest possible performance setting) (block 525). Alternatively, the software driver can delay the current frame rather than playing the previous frame in block 525. If a performance setting exists that can complete the input rendering task in the time remaining until the next Vsync signal (conditional block 520: “Yes”), the software driver operates the processing hardware at the predetermined performance setting (block 530). In one embodiment, the predetermined performance setting is the lowest possible performance setting for completing the input rendering task in the time remaining until the next video sync signal. In another embodiment, the predetermined performance setting is one setting higher than the lowest possible performance setting to provide a tolerance for completing the input rendering task in the time remaining until the next video sync signal. In other embodiments, the tolerance can be increased to two or more settings higher than the lowest possible performance setting.

[0033] If the rendering of the current frame is not finished (conditional block 540: "No"), then method 500 returns to block 505 after some time has passed or after some event (i.e., a change in queue occupancy) is detected. Note that it is possible to add some hysteresis to the loop to prevent performance settings from being changed too frequently. If the rendering of the current frame is finished (conditional block 540: "Yes"), then method 500 ends. Note that method 500 may be performed for each video frame of a video sequence being rendered.

[0034] Referring to FIG. 6, one embodiment of a method 600 for controlling performance settings of processing hardware based on application type is shown. The control unit determines which application is currently being executed by the system (block 605). The control unit then loads a performance setting lookup table (e.g., table 400 of FIG. 4) corresponding to the application (block 610). The control unit then uses the table to select performance settings for the processing hardware based on queue occupancy and the time remaining until the next video sync signal (block 615). If the control unit detects another application being executed by the system (conditional block 620: "Yes"), method 600 returns to block 610. Alternatively, if the given application continues to be executed by the system (conditional block 620: "No"), method 600 returns to block 615. Note that in some cases, a single application can have multiple different performance setting lookup tables. For example, a video game application may have various scenes with different amounts of rendering complexity. In a first scene of an application, the control unit may load a first table, in a second scene the control unit loads a second table, and so on.

[0035] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose processor or a special-purpose processor are contemplated. In various embodiments, such program instructions may be expressed in a high-level programming language. In other embodiments, the program instructions may be compiled from the high-level programming language into binary, intermediate, or other form. Alternatively, program instructions describing the operation or design of hardware may be written. Such program instructions may be expressed in a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog may be used. In various embodiments, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Typically, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0036] It should be emphasized that the above-described embodiments are merely non-limiting examples of embodiments. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. a power management unit, A control unit is provided, The control unit monitoring inputs corresponding to rendering tasks to be performed by one or more processors to render a current video frame; accessing data identifying a performance setting based on the number of rendering tasks and the time remaining until the next video sync signal; In response to the access, obtaining the performance setting from the data; performing the rendering task using the performance settings; and configured to: Power management unit.

2. the control unit is configured to, in response to determining that the rendering task cannot be completed before a next synchronization signal, cause a previous frame to be played. The power management unit of claim 1 .

3. the control unit is configured to operate processing hardware at the lowest possible performance setting in response to determining that the rendering task can be completed before the next synchronization signal while operating at the lowest possible performance setting. The power management unit of claim 1 .

4. The data is stored in a first table including a plurality of entries. The power management unit of claim 1 .

5. the first table corresponds to a given application, and a second table corresponding to a different application stores data specifying performance settings based on a mapping between the number of rendering tasks and the time remaining until the next synchronization signal, the data stored in the second table being different from the data stored in the first table; The power management unit of claim 4.

6. the control unit is configured to update the values ​​stored in the entries of the first table based on operation of the predetermined application. The power management unit of claim 5.

7. The control unit determining which applications are currently running; loading a performance setting lookup table corresponding to the determined application; obtaining the data from the performance setting lookup table to select the performance setting; configured to: The power management unit of claim 1 .

8. Monitoring inputs corresponding to rendering tasks to be performed by one or more processors to render a current video frame; accessing data identifying a performance setting based on a mapping between the number of rendering tasks and the time remaining until the next video sync signal; In response to the access, obtaining the performance setting from the data; and executing the rendering task using the performance settings. method.

9. in response to determining that the rendering task cannot be completed before the next synchronization signal, playing a previous frame.

9. The method of claim 8.

10. In response to determining that the rendering task can be completed before the next synchronization signal while operating at the lowest possible performance setting, the method further includes operating processing hardware at the lowest possible performance setting.

9. The method of claim 8.

11. The data is stored in a first table including a plurality of entries.

9. The method of claim 8.

12. the first table corresponds to a given application, and a second table corresponding to a different application stores data specifying performance settings based on a mapping between the number of rendering tasks and the time remaining until the next synchronization signal, the data stored in the second table being different from the data stored in the first table; The method of claim 11.

13. updating values ​​stored in entries of the first table based on the operation of the predetermined application; 13. The method of claim 12.

14. With multiple tables, Each table corresponds to one or more different applications and one scene of a given application. The method of claim 11.

15. 1. An apparatus comprising: a first processor; a second processor; and a memory for storing program instructions; The program instructions are executable by the first processor to perform the method of any one of claims 8 to 14. Device.

Images