Method and apparatus for automated variable rate shading in the context of a GPU driver

Abstract

A system and method for changing the pixel rate function of a GPU as an optional feature without explicit implementation from within an application is disclosed. User interface (UI) content can be detected in a draw call of an application, and a variable rate shader lookup graph can be generated based on the detected UI content. The variable rate shader lookup graph can be used to increase the pixel rate of 3D content. Additionally or alternatively, other conditions for increasing the pixel rate can be detected, such as using information in an application configuration file, detecting high or low brightness values, detecting motion, and / or detecting temporal anti-aliasing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 233,203, filed on August 13, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein relates to graphics processing units (GPUs). More specifically, the subject matter disclosed herein relates to systems and methods for changing the pixel rate functionality of a GPU as an optional feature without requiring explicit implementation from within an application. Background Art

[0004] With the advent of 4K and 8K digital screens, the number of pixels rendered by the GPU is growing at a dramatic rate. GPUs are increasingly unable to perform the processing required to support new graphics / realistic rendering techniques while rendering all of the pixels at 60-240 frames per second (FPS). To compensate for this large amount of processing, shortcuts exist to reduce the workload of the rendering engine while leaving the image quality subjectively unaffected.

[0005] One shortcut technique is called variable rate shading (VRS). Variable rate shading is a technique that copies a single pixel into blocks of pixels whose sizes are multiples of: 2×1, 1×2, 2×2, 2×4, 4×2, or 4×4. VRS can use a frame buffer mask to dictate which pixels are copied into which block sizes; the pixel rate calculation for each material; or the pixel rate for each drawing. VRS may be set by the application or game when the graphics driver exposes this feature. Application / game writers can choose how to use this feature, choosing the degree of pixel processing savings and the degree of image quality degradation. However, for applications that are no longer under development or maintenance, this feature is too new and will not be taken advantage of by these applications. Summary of the invention

[0006] Example embodiments provide a method for changing a pixel rate in a GPU, wherein the method may include: detecting, by the GPU, user interface (UI) content in a drawing call of an application; generating, by the GPU, a variable rate shader lookup map based on at least one position of the UI content detected in the drawing call; and increasing, by the GPU, the pixel rate of 3D content of the drawing call based on the variable rate shader lookup map. In one embodiment, the method may also include determining, by the GPU, at least one position in the 3D content corresponding to a predetermined brightness value threshold or a predetermined brightness spatial frequency threshold, wherein generating the variable rate shader lookup map may also be based on at least one position in the 3D content corresponding to the predetermined brightness value threshold or the predetermined brightness spatial frequency threshold. In another embodiment, at least one position in the 3D content may include a position having a brightness value less than a predetermined brightness value threshold or a brightness spatial frequency value less than a predetermined brightness spatial frequency threshold. In yet another embodiment, at least one position in the 3D content may include a position having a brightness value greater than a predetermined brightness value threshold or a brightness spatial frequency value greater than a predetermined brightness spatial frequency threshold. In yet another embodiment, the method may further include determining at least one position in the 3D content corresponding to a predetermined motion estimation threshold, wherein generating the variable rate shader lookup graph may also be based on the at least one position in the 3D content corresponding to the predetermined motion estimation threshold. In one embodiment, detecting UI content in the drawing call may include detecting UI content based on a profile of the application. In another embodiment, the method may further include detecting, by the GPU, a post-processing temporal anti-aliasing buffer for the variable rate shader lookup graph or data corresponding to temporal anti-aliasing. In yet another embodiment, detecting UI content in a draw call may include at least one of the following: determining that the draw call comprises a 2D draw call based on two triangles being detected as boxes; determining that the draw call comprises a 2D draw call based on detecting a shader bound to a pipeline of a GPU to be used for a UI; determining that the draw call comprises a 2D draw call based on detecting that the draw call does not use skinning or high dynamic range shading to render the drawing; determining that the draw call comprises a 2D draw call based on determining that the draw call is rendered during the last pass of a frame; determining that the draw call comprises a 2D draw call based on determining that a single texture is indexed into a small equal box size to indicate font retrieval; or determining that the draw call comprises a 2D draw call based on determining that depth testing for the draw call is disabled, the draw call order, or the rendering quadrant of the draw call.

[0007] Example embodiments provide a method of changing a pixel rate in a GPU, wherein the method may include: determining, by the GPU, at least one position in 3D content of a drawing call of an application corresponding to a predetermined luminance value threshold, a predetermined luminance spatial frequency threshold, or a predetermined motion estimation threshold; generating, by the GPU, a variable rate shader lookup graph based on the at least one position corresponding to the predetermined luminance value threshold, the predetermined luminance spatial frequency threshold, or the predetermined motion estimation threshold; and increasing, by the GPU, the pixel rate of the 3D content corresponding to the at least one position corresponding to the predetermined luminance value threshold, the predetermined luminance spatial frequency threshold, or the predetermined motion estimation threshold. In one embodiment, the at least one position in the 3D content corresponds to a position having a luminance value less than a predetermined luminance value threshold, a luminance spatial frequency value less than a predetermined luminance spatial frequency value, or a motion estimation value less than a predetermined motion estimation value threshold. In another embodiment, the at least one position in the 3D content corresponds to a position having a luminance value greater than a predetermined luminance value threshold, a luminance spatial frequency value greater than a predetermined luminance spatial frequency value, or a motion estimation value greater than a predetermined motion estimation value threshold. In yet another embodiment, the method may further include detecting, by the GPU, a post-processing temporal anti-aliasing buffer for the variable rate shader lookup graph or data corresponding to temporal anti-aliasing. In yet another embodiment, the method may further include detecting, by the GPU, UI content in a draw call, wherein generating the variable rate shader lookup graph may further include generating the variable rate shader lookup graph based on at least one position of the UI content detected in the draw call, wherein increasing the pixel rate of the 3D content may further be based on at least one position of the detected UI content. In yet another embodiment, detecting the UI content in the draw call may further include detecting the UI content based on a profile of the application. In one embodiment, detecting UI content in a draw call comprises at least one of: determining that the draw call comprises a 2D draw call based on two triangles being detected as boxes; determining that the draw call comprises a 2D draw call based on detecting a shader bound to a pipeline of a GPU to be used for the UI; determining that the draw call comprises a 2D draw call based on detecting that the draw call does not use skinning or high dynamic range shading to render the drawing; determining that the draw call comprises a 2D draw call based on determining that the draw call is rendered during the last pass of a frame; determining that the draw call comprises a 2D draw call based on determining that a single texture is indexed into a small equal box size to indicate font retrieval; or determining that the draw call comprises a 2D draw call based on determining that depth testing for the draw call is disabled, the draw call order, or the rendering quadrant of the draw call.

[0008] Example embodiments provide a GPU, which may include a graphics pipeline and a controller. The graphics pipeline may be configured to increase the pixel rate of rendered content. The controller may be coupled to the graphics pipeline, and the controller may be configured to: detect at least one position in a UI content in a drawing call of an application or a 3D content of the drawing call corresponding to a predetermined brightness value threshold, a predetermined brightness spatial frequency threshold, or a predetermined motion estimation threshold; generate a variable rate shader lookup map based on at least one position of the UI content detected in the drawing call and at least one position in the 3D content corresponding to the predetermined brightness value threshold, the predetermined brightness spatial frequency threshold, or the predetermined motion estimation threshold; and increase the pixel rate of the corresponding 3D content based on the variable rate shader lookup map. In one embodiment, at least one position in the 3D content may include a position having a brightness value less than a predetermined brightness value threshold, a brightness spatial frequency value less than a predetermined brightness spatial frequency value, or a motion estimation value less than a predetermined motion estimation value threshold. In another embodiment, at least one position in the 3D content may include a position having a brightness value greater than a predetermined brightness value threshold, a brightness spatial frequency value greater than a predetermined brightness spatial frequency value, or a motion estimation value greater than a predetermined motion estimation value threshold. In yet another embodiment, detecting UI content in a drawing call may include detecting UI content based on a profile of the application.In yet another embodiment, the controller may also be configured to detect a post-processing temporal anti-aliasing buffer for a variable rate shader lookup graph or related data corresponding to temporal anti-aliasing. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the following sections, various aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0010] Figure 1 Depicts a typical visual image of different VRS pixel rate changes provided by commercially available GPUs;

[0011] Figure 2 is a functional block diagram of an example embodiment of a GPU including capabilities for automating VRS functions in accordance with the subject matter disclosed herein;

[0012] Figure 3 is a flow chart of an example method for automating a VRS system according to the subject matter disclosed herein;

[0013] Figure 4 Depicting some example 2D detection methods according to the subject matter disclosed herein, which may be part of a 2D detection profile; and

[0014] Figure 5Depicted is an electronic device including a GPU that can increase pixel rate as an optional feature without explicit implementation from within an application in accordance with the subject matter disclosed herein. DETAILED DESCRIPTION

[0015] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, those skilled in the art will appreciate that the disclosed aspects may be practiced without these specific details. In other cases, well-known methods, processes, components, and circuits are not described in detail to avoid obscuring the subject matter disclosed herein.

[0016] References throughout this specification to "one embodiment" or "an embodiment" mean that the specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases with similar meanings) that appear in various places throughout this specification may not necessarily all refer to the same embodiment. In addition, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word "exemplary" means "as an example, instance, or illustration". Any embodiment described herein as "exemplary" should not be interpreted as necessarily preferred or superior to other embodiments. In addition, specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In addition, depending on the context discussed herein, a singular term may include a corresponding plural form, and a plural term may include a corresponding singular form. Similarly, hyphenated terms (e.g., "two-dimensional", "predetermined", "pixel-specific", etc.) may occasionally be used interchangeably with corresponding non-hyphenated versions (e.g., "two-dimensional", "predetermined", "pixel-specific", etc.), and capitalized terms (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock", "rowselect", "pixout", etc.). Such occasional interchangeability should not be considered inconsistent with each other.

[0017] In addition, depending on the context of the discussion herein, singular terms may include corresponding plural forms and plural terms may include corresponding singular forms. It should also be noted that the various figures (including component figures) shown and discussed herein are for illustrative purposes only and are not drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. In addition, if deemed appropriate, reference numerals have been repeated in the accompanying drawings to indicate corresponding and / or similar elements.

[0018] The terms used herein are for the purpose of describing some example embodiments only and are not intended to limit the claimed subject matter. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "comprises" and / or "comprising" specify the presence of the features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] It will be understood that when an element or layer is referred to as being on, "connected to," or "coupled to" another element or layer, it may be directly on, connected to, or coupled to another element or layer, or there may be intermediate elements or layers. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intermediate elements or layers. The same reference numerals always refer to the same element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0020] As used herein, the terms "first", "second", etc. are used as labels for the nouns that precede them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. In addition, the same figure numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such usage is only for simplicity of explanation and convenience of discussion; this does not mean that the construction or architectural details of such components or units are the same in all embodiments, or that such commonly referenced parts / modules are the only way to implement some example embodiments disclosed herein.

[0021] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter belongs. It will be further understood that terms (e.g., those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0022] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware configured to provide the module-related functionality described herein. For example, software may be embodied as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any implementation described herein may include, for example, a single or any combination of assemblies, hardwired circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits. Modules may be collectively or individually embodied as circuits that form part of a larger system, such as, but not limited to, an integrated circuit (IC), a system on a chip (SoC), an assembly, and the like.

[0023] In one embodiment, the subject matter disclosed herein provides a graphics driver that provides automated VRS functionality as an optional feature without explicit implementation from within an application or game. In one embodiment, the subject matter disclosed herein provides a method in a graphics driver to parse existing rendered content and provide VRS to improve performance without a perceptible loss in image quality. Thus, older games that do not have the necessary three-dimensional (3D) engine features may be able to take advantage of the VRS functionality to provide increased frames per second (or for power saving) without unwanted visual display artifacts. In one embodiment, an end user is able to enable / disable / adjust the automated VRS provided herein using, for example, a control panel or configuration menu.

[0024] In another embodiment, the graphics driver identifies a two-dimensional (2D) user interface (UI) portion of a frame during the rendering process and creates a VRS map based on the UI for use in subsequent frames so that the 3D portion of the content hidden by the 2D UI may have a pixel rate that can be maximized. The graphics driver may analyze the immediately preceding frame (frame N-1) or the nearly immediately preceding frame (frame NX) of the 3D content for, for example, one or more of the following conditions: (1) minimal variation in color gradients over an area (mid-tones); (2) dark areas below a luminance value threshold (normal compression techniques); (3) areas of very bright colors above a luminance value threshold (normal compression techniques), areas of luminous spatial frequencies greater than a detectable threshold (normal compression techniques); (4) motion estimation information, which can be used to increase the pixel rate of certain areas because greater motion may make the increased pixel rate variation less noticeable; and (5) detection of temporal anti-aliasing (TAA) to determine where to perform smoothing using prior frame knowledge in order to more aggressively set the VRS pixel rate level.

[0025] The subject matter disclosed herein may also provide a GPU driver that can use an application profile to determine how to isolate 2D UI content from 3D content. Additionally or alternatively, the GPU driver may detect post-processing and create a VRS map (entire frame buffer, or in tiles) by detecting UI overlays and potential areas of the frame buffer where pixel rate changes may have limited visual detection.

[0026] The automated VRS functionality as disclosed herein may be provided by any device (e.g., a mobile application processor (AP), but not limited to a mobile AP), and may be used in any computer system using one or more central processing units (CPUs), GPUs, neural processing units (NPUs), and display processing units (DPUs). The AP may include a GPU (or another engine) capable of VRS changes of pixel rates based on screen space, drawing calls, and / or drawing materials. In addition, the GPU may include hardware improvements to accelerate optical flow / motion estimation based on sequential frame buffer comparisons. The GPU may include a physical implementation of a frame buffer overlay, and the overlay may be used to indicate a 2D UI screen space position for use by automated VRS. The GPU may be enabled from a configuration option provided by the graphics driver so that applications that do not currently use VRS may use VRS. A neural network (NN) may also be trained to provide various graphs and parameters for Tier 1 VRS and Tier 2 VRS. Every nth frame may be updated, and VRS may be disabled to use ground truth information. As used herein, the term "Tier2VRS" refers to a VRS technique that uses a screen space lookup map to define which pixels are drawn at a specific pixel rate, while the term "Tier 1VRS" refers to a VRS technique that has a much finer granularity than Tier 2VRS and is typically defined by the application where the pixel rate can be changed for each draw call. The subject matter disclosed herein is applicable to both Tier 1VRS and Tier 2VRS. For Tier 1VRS, an application profile can be used to define which drawings are degraded.

[0027] The subject matter disclosed herein also provides a system and method for automating VRS. In one embodiment, the system may include a GPU that supports Tier 2 VRS functionality and can use one or more pixel rate lookup maps to determine pixel output. The pixel rate lookup map can be generated from metadata within a graphics device driver or based on analysis of an output pixel buffer. The metadata can be used to generate a pixel rate lookup for UI isolation within the driver through one or more locations of UI elements that indicate that VRS may increase the screen space of the pixel output of each shader output because the UI will cover these pixel areas. The metadata can also be used by using a temporal anti-aliasing (TAA) buffer as a potential pixel rate lookup map to detect TAA within the application. Analysis of the output pixel buffer may include, but is not limited to, analysis of the following: a luminance value threshold (high or low luminance) or a luminance spatial frequency threshold as an indicator of an area on the frame buffer where the pixel rate may be increased due to known limitations of human perception; and a motion estimation comparison of multiple sequential frames to indicate potential blur regions (which may indicate areas where the pixel rate may be increased).

[0028] The graphics driver can be configured to use VRS automatically and can be assisted by an application or game profile that includes hints that can indicate to the application how to identify UI elements, render targets / buffers, and image dependencies relative to the game. The application can use the render targets / buffers to calculate temporal anti-aliasing (for motion detection) and application-defined motion estimation related resources (such as vector maps for optical flow). Image dependencies relative to the game can be used to directly indicate the VRS pixel map.

[0029] Another example embodiment provides a method for automatically applying a VRS function by a GPU to control pixel output of a shader that supports high-level VRS, wherein the method may include detecting that the VRS function is enabled, generating a VRS pixel rate lookup map from metadata inside a graphics device driver or from analysis of an output pixel buffer, and controlling the shader pixel output based on the contents of the VRS pixel rate lookup map.

[0030] Yet another example embodiment may include generating a VRS pixel rate lookup map from an analysis of an output pixel buffer using one or more of luma value or luma spatial frequency thresholds and / or motion estimation comparisons of multiple sequential frames. Luma value thresholds and luma spatial frequency thresholds that may be used for high or low luma may be used as indicators of areas on the frame buffer where pixel rate may be reduced due to known limitations of human perception. Motion estimation comparisons of multiple sequential frames may be used to indicate potential blurry areas where increased pixel rate may be applied.

[0031] Another example embodiment may provide a method for generating a VRS pixel rate lookup map from metadata within a graphics device driver, including using UI isolation within the graphics device driver and / or using application detection of temporal anti-aliasing (TAA). UI isolation may involve one or more locations of UI elements that indicate screen space where VRS may reduce pixel output for each shader output due to the UI covering these pixel areas. In addition, detection of temporal anti-aliasing (TAA) within the application may involve a post-processing TAA buffer that may be used for the VRS pixel rate lookup map.

[0032] Yet another example embodiment may provide a graphics driver for a GPU that automatically uses VRS based on an application profile, which may include hints indicating how to identify UI elements for isolation, render targets / buffers used by the application for computational temporal anti-aliasing (i.e., motion detection), application-defined motion estimation-related resources (such as one or more vector maps for optical flow), and image dependencies for the game relative to directly indicating a VRS pixel map.

[0033] Figure 1 Depicted are typical visual images of different VRS pixel rate variations provided by commercial GPUs. The image at 101 is a default 1x1 pixel image. The image at 102 is a 2x1 VRS pixel rate image, and the image at 103 is a 1x2 VRS pixel rate image. The image at 104 is a 2x2 VRS pixel rate image. The image at 105 is a 4x2 VRS pixel rate image, and the image at 106 is a 2x4 VRS pixel rate image. The image at 107 is a 4x4 pixel rate image.

[0034] Figure 22 is a functional block diagram of an example embodiment of a GPU 200 including capabilities for automated VRS functionality in accordance with the subject matter disclosed herein. The GPU 200 may include a controller 201 and a graphics pipeline 202. The controller 201 may interface with a GPU driver 203 and may be responsive to the GPU driver 203. An application / game 204 may interface with the GPU driver 203 for rendering on a display (not shown) coupled to the GPU. The graphics pipeline 202 may include functionality for detecting 2D and 3D drawings to provide automated VRS functionality as described herein. The graphics pipeline 202 may include, but is not limited to, one or more vertex shaders, hull shaders, geometry shaders, perspective transformers, and / or pixel shaders, one or more clipping and culling stages, one or more rasterization stages, and one or more depth testing stages, all of which are not specifically shown because the pipeline configuration may be specific to the GPU embodiment. The graphics pipeline 202 may also include a primary frame buffer 205 for 3D content and an auxiliary buffer or VRS lookup map 206 for locations in the current rendered frame for one or more 2D UIs. The primary frame buffer 206 and the VRS lookup map may be combined in a buffer 207 for output to a display (not shown). The GPU 200 may include a neural processing unit (NPU) 208 and / or a display signal processor (DSP) 209 to perform and / or assist in identifying UI content from 3D content. Additionally or alternatively, an application specific integrated circuit (ASIC) 210 may be coupled to the GPU 200 to perform and / or assist in identifying UI content from 3D content.

[0035] The various components and functional capabilities of the GPU 200 may be provided by one or more modules. Some embodiments of the GPU 200 may be implemented on a mobile application processor, but it should be understood that the implementation is not limited thereto and may be implemented in other contexts, for example, a computer system that may utilize a subset of a central processing unit (CPU), a GPU, an NPU, and / or a DPU. In addition, some embodiments may use the GPU 200 with a NN model to combine or mix the output of an application with a VSR function. The automated VRS functionality of the GPU 200 may include an on / off feature that can be selected by the user and / or OEM when the VRS feature should be activated and when it should not be activated. The level of VRS may also be selectable to provide a desired level of image quality.

[0036] The VRS functionality can be selectively enabled based on different detectable image conditions. For example, a VRS increased pixel rate can be selected for pixels below a 2D UI element (i.e., UI overlay detection). Pixels with low brightness or very high brightness can be selected for a VRS increased pixel rate (brightness detection). That is, a frame buffer with relatively dark brightness or relatively bright brightness can be selected for a VRS increased pixel rate. Brightness detection can also be used for spatial frequency conditions of pixels in an area (i.e., very rapid changes) and / or motion estimation conditions (i.e., motion blur).

[0037] When UI overlay detection is enabled, the graphics driver detects UI-related draw calls. The partition can attach a meta tag to all draw calls that are considered UI to mark the UI pixel locations in the secondary render target. The alpha value of the UI can also be recorded to identify relatively more transparent locations of the UI so that the original pixel rate will not change much. Before runtime, GPU vendors can generate configuration files to identify draw call partitions to isolate 2D and 3D based draw calls. These UI locations can indicate areas with higher pixel shader rates (larger than 1x1) because these areas will have 2D content that occludes the original pixels.

[0038] When brightness detection is enabled, the fully rendered output frame can be post-processed (as a whole or in tiles) by the rendering GPU (or by additional available accelerators (NPU / DSP / CPU)) for low brightness areas, high brightness areas, and / or luminous spatial frequency areas with very fast brightness changes that appear mixed.

[0039] For cases where motion estimation (motion blur) can be calculated by the application, the graphics driver can intercept a post-processing buffer that tracks pixel velocity to use as a potential pixel rate lookup map. Motion blur can typically be performed by measuring the movement of the "camera" and can be combined with the velocity of objects moving in the 3D scene. Pixel-velocity information can typically be stored in a render target / texture for use in post-processing effects.

[0040] Thus, detection for application VRS can be simplified since pixel-velocity maps are typically generated as a post-processing step. For example, the driver can check compute shader resources based on this common method for writing or reading from pixel velocity maps. In addition, hints based on application profiles can be provided to the driver based on the application header on how to find the pixel-velocity map. When found, the pixel-velocity texture will provide a potential VRS map for automated VRS provided by the driver. Hints can also be provided through offline profiling.

[0041] When motion blur cannot be intercepted from the application, a comparison of the current frame N and the previous frame N-1 can be performed based on the original position without metadata that the application can use. That is, computer vision methods for detecting motion in sequential images can be used. Such computer vision methods may include, but are not limited to, methods for measuring optical flow; methods for detecting motion segments in sequential images / videos; and / or other visual I-frame prediction methods performed by video compression codecs.

[0042] If motion estimation can be detected by applying a level pixel velocity map or by sequential image correlation, the magnitude of the motion tensor may indicate a proportional blur that can be used by increasing the pixel rate of the VRS.

[0043] For all tiles of the VRS map whose pixel rate is increased from the native (1×1) pixel rate, it can be expected that the orientation of the VRS pixel block can be consistent with the spatial similarity obtained by brightness detection. The orientation when it is related to the VRS pixel block is not isotropic (e.g., 2×2 or 4×4 pixel blocks), but anisotropic (e.g., 2×1, 1×2, 4×2 and 2×4 pixel blocks).

[0044] UI / brightness / motion estimation operations can be trained into a NN model that provides similar behavior to also create a lookup graph for Tier1 / Tier2 VRS based on the pixel location in the render target (frame buffer). Inference of the NN model can be computed using an NPU / GPU / CPU or other accelerator.

[0045] The calculation of brightness / motion estimation may be expected to be processed once every N frames. During the frames where brightness / motion estimation is processed, the VRS function may be disabled to describe the rendered image with the highest accuracy. For motion estimation, the VRS function may be disabled more frequently to compare the (current) Nth frame with the (previous) N-1th frame using the ground truth image. In one embodiment, the VRS samples may be reduced in proportion to the motion.

[0046] Figure 3300 is a flow chart of an example method 300 of an automated VRS system according to the subject matter disclosed herein. The method 300 begins at 301 during rendering setup. At 302, it is determined whether the automated VRS feature is enabled. If so, the flow continues to 303, where the pipeline is set up for the automated VRS feature. The setup of the pipeline can be based on a profile 304 of a running application for identifying and isolating 2D UIs. Components of the application profile 304 can include one or more signatures (or features) of vertex data, such as vertex count and position data; one or more signatures of textures bound to a draw call; one or more render pass signatures / pipeline signatures for UI draw calls; one or more depth / stencil test states when drawing; one or more shader signatures and / or uniform data indicating UI drawing; and / or one or more locations of a render pass in a command buffer containing UI drawings in a frame.

[0047] The flow continues to 305, where two frame buffers are created and bound to the pipeline. That is, a primary frame buffer and a secondary frame buffer are created and bound to the pipeline. At 306, rendering begins. At 307, UI detection is performed, and any UI position data can be stored in the VRS lookup map at 313.

[0048] Middleware engines can be the most popular engines for current generation graphics content and can provide a clean separation between scene art and construction and the graphics application programming interface (API) calls that drive the GPU. The separation of art / construction and API can often isolate 3D content from 2D overlays in source code space and can be seen by the driver in the API flow through the separation of rendering passes.

[0049] The UI (despite the potential Z-rejection benefit that is always on top) can typically be rendered in the last rendering pass, with the expectation that the UI does not cover many pixels, otherwise interesting parts of the scene may be obscured by relatively uninteresting status indicators. The UI can be identified based on the drawing not using high dynamic range (HDR) shading, since most UIs are flat texture maps. In addition, texture sample pattern detection in the GPU shader compiler for font textures, which are accessed in a very specific sub-tiling pattern to index each character of the font map, can be used to identify the UI. The UI can also be detected by detecting that depth testing is disabled. That is, because the UI is meant to be forced to be visible and the back-to-front order is indicated offline, depth testing may not be required during its rendering.

[0050] UIs can also be detected by detecting changes in drawing parameter patterns. Standard post-processing graphics applications may render the main 3D geometry with seemingly random drawing parameters, followed by one or more draws containing four vertices that form quads generated from triangle strips. Such quads can be used to cover the entire render target to apply post-processing operations for each pixel. Since UIs are not typically operated on a pixel-by-pixel basis, nor are UIs post-processed, it can be expected that UIs may follow a post-processing pass and may contain more than four vertices.

[0051] In one embodiment, the UI detection at 307 can be based on Figure 3 The UI detection configuration file 308 in. Figure 4 Some example 2D detection methods that may be part of 2D detection profile 308 in accordance with the subject matter disclosed herein are depicted. Figure 4 The detection profile items described in are exemplary only and are not a comprehensive list. Figure 4 Item 308a in relates to changes in drawing parameter patterns that can be used to identify 2D UI overlays. For example, a post-processed graphics application may render primary 3D geometry with apparently random drawing parameters, followed by one or more drawings containing four vertices that form quadrilaterals generated by triangle strips. The quadrilaterals may be used to cover part or all of the render target to apply post-processing operations for each pixel. Since UIs may not be pixel-by-pixel operations, and / or they may not be processed by post-processing, they may follow a post-processing pass and may contain more than four vertices and thus be detected as 2D UI overlays.

[0052] Item 308b involves detecting 2D UI overlays by identifying specific shaders as bound to a pipeline that is used only for UI rendering. For example, these specific shaders may not use skinning or high dynamic range shading. Drawings that do not use HDR shading may be detected as 2D UI content because the UI is a flat texture map.

[0053] Item 308c involves detecting a 2D UI that, despite the potential z-rejection benefit, is on top, as it is typically rendered during the last rendering pass of a frame, where it is desirable that the UI should not cover many pixels, otherwise interesting parts of the scene may be obscured by relatively uninteresting status indicators. Furthermore, it is typically the last drawing in a frame, since it is always on top and will always overwrite competing pixels.

[0054] Item 308d relates to font textures that can be detected by indexing into a single texture with small equal sized boxes. A 2D UI overlay can be identified as a texture sample pattern in the GPU shader compiler of a font texture that can be accessed in a specific sub-tiling pattern to index into one or more characters of a font map.

[0055] Item 308e relates to disabled depth testing, which can be used to identify 2D UI overlays, since UIs are typically forced to be visible and indicated offline in a back-to-front order, so depth testing is not used during their rendering.

[0056] return Figure 3 , at 309, it is determined whether the frame drawing is the Nth frame drawing. If so, the flow continues to 310, at which pixels are rendered into the primary frame buffer without using VRS. At 311, calculations are performed based on brightness and motion estimation. The calculations may be performed by a GPU, NPU, DSP, and / or CPU. At 312, the resulting data from the brightness and motion estimation calculations are merged into the VRS lookup map.

[0057] If it is determined at 309 that the frame drawing is not the Nth drawing, flow continues to 314 where the current VRS lookup map at 312 is applied to the pixel drawing. From 314, flow continues to 315 where the output is sent to a display.

[0058] If in Figure 3 If it is determined at 302 in that the automatic VRS function is not enabled, the flow continues to 316, at which a single frame buffer is created and bound to the pipeline. At 317, rendering begins as usual. The flow continues to 315, at which the output is sent to the display.

[0059] For a particular application title, pre-arranged hints may be compiled into an application profile (e.g., application profile 304) to improve 2D UI isolation. This may not be necessary if the method described in conjunction with UI detection at 307 is deemed sufficient. If no suitable techniques for 2D UI isolation are provided for a particular application and the results of isolating the UI are poor, the profile may suggest that the automated VRS function for UI isolation may be disabled entirely, in which case, Figure 3 The flow in will be from 303 to 316.

[0060] Figure 5An electronic device 500 including a GPU that can increase pixel rate as an optional feature without explicit implementation from within an application is depicted in accordance with the subject matter disclosed herein. The electronic device 500 may include a controller (or CPU) 510, an input / output device 520 (e.g., but not limited to a keypad, keyboard, display, touch screen display, camera, and / or image sensor), a memory 530, an interface 540, a GPU 550, an imaging processing unit 560, and a neural processing unit 570, which are coupled to each other via a bus 580. The controller 510 may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, etc. The memory 530 may be configured to store command codes or user data to be used by the controller 510.

[0061] According to the subject matter disclosed herein, the electronic device 500 and various system components of the electronic device 500 may include an image processing unit 560, which may include a GPU that can increase the pixel rate as an optional feature without explicit implementation from within the application. The GPU 550 may also be able to increase the pixel rate as an optional feature without explicit implementation from within the application. The interface 540 may be configured to include a wireless interface that is configured to send data to or receive data from a wireless communication network using RF signals. The wireless interface 540 may include, for example, an antenna. The electronic system 500 can also be used for communication interface protocols of communication systems, such as but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), North American digital communications (NADC), extended time division multiple access (E-TDMA), wideband CDMA (WCDMA), CDMA2000, WiFi, municipal WiFi (Muni WiFi), Bluetooth, digital enhanced cordless telecommunications (DECT), wireless universal serial bus (WirelessUSB), orthogonal frequency division multiplexing with fast low latency access with seamless switching (Flash-OFDM), IEEE 802.20, general packet radio service (GPRS), iBurst, wireless broadband (WiBro), WiMAX, WiMAX-Advanced, universal mobile telecommunications service-time division duplex (UMTS-TDD), high speed packet access (HSPA), evolution data optimized (EVDO), long term evolution-advanced (LTE-Advanced), multi-channel multipoint distribution service (MMDS), fifth generation wireless (5G), sixth generation (6G) wireless, etc.

[0062] The subject matter and the embodiments of the operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device to be executed by a data processing device. The computer storage medium may be or be contained in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be contained in one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices). Furthermore, the operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0063] Although this specification may contain many specific implementation details, the implementation details should not be interpreted as a limitation on the scope of any claimed subject matter, but rather as a description of features specific to a particular embodiment. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0064] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0065] Thus, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

[0066] As will be appreciated by those skilled in the art, the innovative concepts described herein can be modified and varied over a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings described above, but instead is defined by the appended claims.

Claims

1. A method for changing a pixel rate in a graphics processing unit (GPU), the method comprising: The GPU detects the UI content in the application's drawing calls; generating, by the GPU, a variable rate shader lookup graph based on at least one location of the UI content detected in the draw call; and A pixel rate of 3D content of a draw call is increased by a GPU based on a variable rate shader lookup graph, wherein the UI content is automatically detected by the GPU and variable rate shading for the application is automatically enabled within the GPU.

2. The method according to claim 1 , further comprising determining, by the GPU, at least one position in the 3D content corresponding to a predetermined brightness value threshold or a predetermined brightness spatial frequency threshold, in, Generating the variable rate shader lookup graph is also based on at least one location in the 3D content corresponding to a predetermined luminance value threshold or a predetermined luminance spatial frequency threshold.

3. The method according to claim 2, wherein: The at least one position in the 3D content includes a position having a luminance value less than a predetermined luminance value threshold or a luminance spatial frequency value less than a predetermined luminance spatial frequency threshold.

4. The method according to claim 2, wherein: The at least one position in the 3D content includes a position having a luminance value greater than a predetermined luminance value threshold or a luminance spatial frequency value greater than a predetermined luminance spatial frequency threshold.

5. The method of claim 1 , further comprising determining at least one position in the 3D content corresponding to a predetermined motion estimation threshold, and in, Generating the variable rate shader lookup graph is also based on at least one location in the 3D content that corresponds to a predetermined motion estimation threshold.

6. The method according to claim 1, wherein: Detecting UI content in the draw call includes detecting the UI content based on a configuration file of the application.

7. The method according to claim 1, further comprising: Post-processing temporal anti-aliasing buffers or data corresponding to temporal anti-aliasing detected by the GPU for use with variable rate shader lookup graphs.

8. The method according to claim 1, wherein: Detecting UI content in a drawing call includes at least one of the following: Determining that the draw call includes a 2D draw call based on the two triangles being detected as boxes; Determining that the draw calls include 2D draw calls based on detecting a shader bound to a pipeline of a GPU to be used for the UI; Determining that the draw call includes a 2D draw call based on detecting that the draw call does not use skinning or high dynamic range shading to render the drawing; Based on determining that the draw calls are rendered during a last pass of the frame, determining that the draw calls include 2D draw calls; determining that the draw calls include 2D draw calls based on determining that a single texture is indexed into a small equal box size to indicate font retrieval; or The draw call is determined to include a 2D draw call based on determining that depth testing for the draw call is disabled, the draw call order, or a rendering quadrant of the draw call.

9. A method for changing a pixel rate in a graphics processing unit (GPU), the method comprising: Determining, by the GPU, at least one location in the 3D content of the draw call of the application that corresponds to a predetermined luminance value threshold, a predetermined luminance spatial frequency threshold, or a predetermined motion estimation threshold; generating, by the GPU, a variable rate shader lookup map based on at least one position corresponding to a predetermined luma value threshold, a predetermined luma spatial frequency threshold, or a predetermined motion estimation threshold; The GPU detects the UI content in the drawing call; as well as A pixel rate of 3D content corresponding to at least one position corresponding to a predetermined luma value threshold, a predetermined luma spatial frequency threshold, or a predetermined motion estimation threshold is increased by the GPU, wherein the UI content is automatically detected by the GPU and variable rate shading for the application is automatically initiated within the GPU.

10. The method according to claim 9, wherein: At least one position in the 3D content corresponds to a position having a luminance value less than a predetermined luminance value threshold, a luminance spatial frequency value less than a predetermined luminance spatial frequency threshold, or a motion estimation value less than a predetermined motion estimation value threshold.

11. The method according to claim 9, wherein: The at least one position in the 3D content corresponds to a position having a luminance value greater than a predetermined luminance value threshold, a luminance spatial frequency value greater than a predetermined luminance spatial frequency value, or a motion estimation value greater than a predetermined motion estimation value threshold.

12. The method according to claim 9, further comprising: Post-processing temporal anti-aliasing buffers or data corresponding to temporal anti-aliasing detected by the GPU for use with variable rate shader lookup graphs.

13. The method according to claim 9, further comprising detecting, by the GPU, user interface (UI) content in the drawing call, in, Generating the variable rate shader lookup graph further includes generating the variable rate shader lookup graph based also on at least one location of the UI content detected in the draw call, and In which increasing the pixel rate of the 3D content is further based on at least one detected position of the UI content.

14. The method according to claim 13, wherein: Detecting UI content in the draw call includes detecting the UI content based on a configuration file of the application.

15. The method according to claim 13, wherein: Detecting UI content in a drawing call includes at least one of the following: Determining that the draw call includes a 2D draw call based on the two triangles being detected as boxes; Determining that the draw calls include 2D draw calls based on detecting a shader bound to a pipeline of a GPU to be used for the UI; Determining that the draw call includes a 2D draw call based on detecting that the draw call does not use skinning or high dynamic range shading to render the drawing; Based on determining that the draw calls are rendered during a last pass of the frame, determining that the draw calls include 2D draw calls; determining that the draw calls include 2D draw calls based on determining that a single texture is indexed into a small equal box size to indicate font retrieval; or The draw call is determined to include a 2D draw call based on determining that depth testing for the draw call is disabled, the draw call order, or a rendering quadrant of the draw call.

16. A graphics processing unit (GPU), comprising: The graphics pipeline is configured to increase the pixel rate of rendered content; as well as A controller coupled to the graphics pipeline, the controller being configured to: Detecting at least one position in user interface UI content in a drawing call of the application or in 3D content of the drawing call that corresponds to a predetermined brightness value threshold, a predetermined brightness spatial frequency threshold, or a predetermined motion estimation threshold, generating a variable rate shader lookup graph based on at least one position of the UI content detected in the draw call and at least one position in the 3D content corresponding to a predetermined luminance value threshold, a predetermined luminance spatial frequency threshold, or a predetermined motion estimation threshold, and A pixel rate of corresponding 3D content is increased based on a variable rate shader lookup graph, wherein the UI content is automatically detected by a GPU and variable rate shading for the application is automatically enabled within the GPU.

17. The GPU of claim 16, wherein: The at least one position in the 3D content includes a position having a luminance value less than a predetermined luminance value threshold, a luminance spatial frequency value less than a predetermined luminance spatial frequency value, or a motion estimation value less than a predetermined motion estimation value threshold.

18. The GPU of claim 16, wherein: The at least one position in the 3D content includes a position having a luminance value greater than a predetermined luminance value threshold, a luminance spatial frequency value greater than a predetermined luminance spatial frequency value, or a motion estimation value greater than a predetermined motion estimation value threshold.

19. The GPU of claim 16, wherein: Detecting UI content in the draw call includes detecting the UI content based on a configuration file of the application.

20. The GPU of claim 16, wherein: The controller is further configured to detect a post-processing temporal anti-aliasing buffer for a variable rate shader lookup graph or related data corresponding to temporal anti-aliasing.

Images