Super-resolution upscaling

Abstract

A first frame of the video stream is obtained that is rendered at a first resolution. A second frame of the video stream is also obtained that is upscaled to a higher second resolution. The first plurality of pixels is upscaled to the second resolution. The upsampling generates upsampled color data for the first plurality of pixels that are upsampled. The upsampled color data is accumulated with a second set of color data associated with the second plurality of pixels defining the second frame to generate final color data for the upsampled first plurality of pixels. Color data of the second set of color data associated with a pixel lock contributes more color data to the final color data than corresponding color data of the upsampled color data. The upsampled first plurality of pixels, together with the final color data, are stored as an upscaled frame representing the first frame at the second resolution.

Description

Background Art

[0001] Generally, a spatial upscaler implemented in a graphics pipeline receives frames rendered at a resolution lower than the native resolution in order to improve performance, and upsamples these frames to match the native resolution of the display. Although integration is easy, the spatial upscaler also has drawbacks. For example, typically, conventional spatial upscaling approaches require a high-quality anti-aliased source image. Therefore, games that do not use anti-aliasing must implement anti-aliasing, which requires more time to integrate the conventional approach. Furthermore, when the implementation of anti-aliasing is insufficient, conventional spatial upscalers often produce low-quality upscaled outputs. Also, the upscaling quality depends on the input source resolution. When the source resolution is extremely low, the information extracted to reproduce fine details is insufficient, which newly generates artifacts such as shimmering and insufficient edge reconstruction.

Summary of the Invention

Means for Solving the Problems

[0002] This disclosure will be more deeply understood by referring to the following embodiments individually or in combination, and many of its features and advantages will become apparent to those skilled in the art.

[0003] Embodiment 1: A method in an acceleration processing device, comprising: The process involves obtaining a first frame of a video stream rendered at a first resolution, wherein the first frame is defined by a first plurality of pixels associated with a first set of color data; obtaining a second frame of a video stream upscaled to a second resolution higher than the first resolution, wherein the second frame is defined by a second plurality of pixels associated with a second set of color data; and upsampling the first plurality of pixels to the second resolution, wherein upsampling generates upsampled color data for the upsampled first plurality of pixels based on the first set of color data; accumulating the upsampled color data with the second set of color data to generate final color data for the upsampled first plurality of pixels, wherein the color data of the second set of color data associated with pixel locks contributes to the final color data more than the corresponding color data of the upsampled color data. This includes storing the first upsampled pixels, along with the final color data, as an upscaled frame representing the first frame at the second resolution.

[0004] Example 2: The method described in Example 1 is Based on the first set of color data, the exposure value of the first frame is determined, This further includes determining the current brightness of the first frame based on a first set of color data.

[0005] Example 3: In the method described in Example 2, determining the exposure value includes downsampling the first set of color data to store the exposure value at a 1x1 mipmap level, and determining the current luminance includes downsampling the first set of color data to store the current luminance at a mipmap level with half the resolution.

[0006] Example 4: The method described in any of Examples 1 to 3 is, The process further includes locking one or more pixels and converting the above first set of color data from a first color space to a second color space before upsampling the first set of pixels.

[0007] Example 5: The method described in any of Examples 1 to 4 is The process involves expanding the depth values ​​in the depth buffer of the first frame to generate an expanded depth buffer, expanding the motion vectors in the motion vector buffer of the first frame to generate an expanded motion vector buffer, and estimating the depth values ​​of each pixel in the second frame for each of the first plurality of pixels based on the expanded depth buffer and the expanded motion vector buffer. This further includes storing the estimated depth value in a previous depth buffer.

[0008] Example 6: The method described in Example 5 is The process further includes generating a discocclusion mask for each pixel of a first plurality of pixels based on the expanded depth buffer, the expanded motion vector buffer, and the previous depth buffer, wherein the discocclusion mask indicates the amount by which the pixel was deoccluded in the second frame.

[0009] Example 7: The method described in any of Examples 1 to 6 is, For each of the first of a plurality of pixels, the amount of change in the shading of the pixel between the first frame and the second frame is determined, The first further includes generating shading change data for each pixel of a plurality of pixels based on the amount of shading change determined for that pixel.

[0010] Example 8: In the method described in Example 7, Determining the amount of shading change for the above pixel involves comparing the brightness of the pixel at the time the lock was generated with a shading change threshold, depending on whether the pixel is locked. This includes comparing the current brightness of a pixel with the historical brightness information associated with the second frame, depending on whether the pixel is unlocked.

[0011] Example 9: In the method described in any of Examples 1 to 8, Upsampling the above-mentioned first plurality of pixels to the above-mentioned second resolution includes applying a Lanczos resampling kernel to the vicinity of each of the above-mentioned first plurality of pixels, centered on that pixel.

[0012] Example 10: The method described in Example 9 is The method further includes generating a color space bounding box for each of the first plurality of pixels based on the neighborhood of the pixels centered on the aforementioned pixel.

[0013] Example 11: The method according to any of Examples 1 to 10 further includes reprojecting a second set of color data onto the first frame to generate the reprojected color data.

[0014] Example 12: In the method described in Example 11, Reprojecting the second set of color data mentioned above is Sampling motion vectors from the motion vector buffer associated with the first frame, This includes applying the above-sampled motion vector to a second set of color data from the second frame.

[0015] Example 13: The method described in Example 11 is The process further includes reprojecting the set of pixel locks associated with the above-mentioned second set of pixels to generate reprojected pixel locks.

[0016] Example 14: The method described in Example 11 is For each of the upsampled first plurality of pixels, the final historical color value is determined based on the reprojected color data of at least the second frame.

[0017] Example 15: In the method described in Example 14, determining the final historical color value is: By applying the deocclusion mask to the reprojected color data, it is possible to determine one or more pixels that have been deoccluded in the second set of pixels, and to reduce the influence of the color data from the reprojected color data associated with the one or more pixels that have been determined to be deoccluded on the final historical color value. This includes reducing the influence of the color data from the reprojected color data based on the distance of the color data from the color space bounding box generated for the first plurality of upsampled pixels.

[0018] Example 16: In the method described in Example 14, Accumulating the upsampled color data with a second set of color data includes accumulating the upsampled color data of each pixel of the first set of upsampled pixels with the final historical color value.

[0019] Example 17: An acceleration processing device comprising a graphics processing pipeline that performs the method described in any of Examples 1 to 16.

[0020] This disclosure can be better understood by reference to the accompanying drawings, and many of its features and advantages will become apparent to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical items.

Brief Description of the Drawings

[0021] [Figure 1] FIG. 8 is a block diagram of an exemplary device for implementing the super-resolution upscaling technology described herein according to some embodiments. [Figure 2] FIG. 11 is a more detailed block diagram of the exemplary device of FIG. 1 showing further details related to the execution of processing tasks in one or more acceleration processing devices according to some embodiments. [Figure 3] FIG. 14 is a block diagram of the graphics pipeline of FIG. 2 according to some embodiments. [Figure 4] FIG. 17 is an integrated diagram of the graphics pipeline related to the super-resolution upscaler according to some embodiments. [Figure 5] FIG. 20 is a block diagram showing a more detailed view related to the super-resolution upscaler of FIG. 4 according to some embodiments. [Figure 6] FIG. 23 is a block diagram showing various stages / paths of the super-resolution upscaler of FIG. 5 according to some embodiments. [Figure 7] FIG. 26 is a diagram showing the downsampling technology of the automatic exposure component of the super-resolution upscaler of FIG. 5 according to some embodiments. [Figure 8] FIG. 29 is a diagram showing an exemplary post-expansion depth value and motion vector construction process executed by the reconstruction expansion component of the super-resolution upscaler of FIG. 5 according to some embodiments. [Figure 9] FIG. 32 is a block diagram showing various sub-stages of the reprojection accumulation stage of the super-resolution upscaler of FIG. 5 according to some embodiments. [Figure 10]This figure shows an exemplary resampling process performed by the upsampling component of the super-resolution upscaler shown in Figure 5, according to several embodiments. [Figure 11] This figure shows an exemplary color bounding box construction process performed by the upsampling component of the super-resolution upscaler shown in Figure 5, according to several embodiments. [Figure 12] This figure shows an exemplary color data reprojection process performed by the reprojection component of the super-resolution upscaler shown in Figure 5, according to several embodiments. [Figure 13] This figure shows an exemplary color correction process performed by the color corrector component of the super-resolution upscaler shown in Figure 5, according to several embodiments. [Figure 14] This figure shows an exemplary filter performed by the image sharpening component of the super-resolution upscaler in Figure 5, according to several embodiments. [Figure 15] This flowchart illustrates an overall method for spatially upscaling rendered frames of a video stream by reconstructing high-resolution images representing the rendered frames using temporal feedback, according to several embodiments. [Figure 16] This flowchart illustrates another overall method for spatially upscaling rendered frames of a video stream by reconstructing high-resolution images representing the rendered frames using temporal feedback, according to several embodiments. [Figure 17] This flowchart illustrates another overall method for spatially upscaling rendered frames of a video stream by reconstructing high-resolution images representing the rendered frames using temporal feedback, according to several embodiments. [Figure 18] This flowchart shows a more detailed method of the reprojection accumulation process shown in block 1616 of Figure 17, according to several embodiments. [Figure 19] This flowchart shows a more detailed method of the reprojection accumulation process shown in block 1616 of Figure 17, according to several embodiments. [Figure 20] This is a flowchart illustrating the overall method of the pixel locking process, as shown according to several embodiments. [Figure 21] This flowchart shows a more detailed method of the pixel locking process, as shown in block 1614 of Figure 17, according to several embodiments. [Figure 22] A more detailed method of reprojection shown in block 1806 of Figure 18, according to several embodiments, and a flowchart showing the lock update process shown in block 1808 of Figure 18. [Modes for carrying out the invention]

[0022] As described below, the exemplary approach can improve upscaling by using temporal feedback to reconstruct high-resolution images while maintaining or improving image quality compared to native rendering. This approach makes "practical execution" of costly rendering operations such as hardware ray tracing possible.

[0023] Figure 1 shows an exemplary device 100 capable of implementing one or more functions described herein, such as a super-resolution upscaler 332 (Figure 3). Device 100 includes, for example, a computer, a game device, a portable device, a set-top box, a television, a mobile phone, a tablet computer, a wearable computing device, and the like. In one or more embodiments, device 100 includes one or more processors 102, memory 104, a storage device 106, an input device 108, and an output device 110. In at least some embodiments, device 100 includes one or more input drivers 112 and one or more output drivers 114. It should be understood that device 100 may include additional components not shown in Figure 1.

[0024] In one or more embodiments, the processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU located on the same die, or one or more processor cores, each of which may be a CPU or a GPU. In at least some embodiments, the CPU includes one or more single-core or multi-core CPUs. As various alternatives, the memory 104 may be located on the same die as the processor 102 or separately from the processor 102. The memory 104 may include, for example, random access memory (RAM), dynamic RAM, or volatile or non-volatile memory such as a cache.

[0025] Storage 106 includes, for example, fixed or removable storage such as a hard disk drive, solid-state drive, optical disc, or flash drive. Input devices 108 include, for example, a keyboard, keypad, touchscreen, touchpad, detector, microphone, accelerometer, gyroscope, biometric scanner, or network connectivity (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals). Output devices 110 include, for example, a display, speaker, printer, haptic feedback device, one or more lights, antenna, or network connectivity (e.g., a wireless local area network card for transmitting and / or receiving wireless IEEE 802 signals).

[0026] The input driver 112 communicates with the processor 102 and the input device 108, enabling the processor 102 to receive input from the input device 108. The output driver 114 communicates with the processor 102 and the output device 110, enabling the processor 102 to send output to the output device 110. It should be noted that the input driver 112 and the output driver 114 are optional components, and the device 100 will function similarly even if the input driver 112 and the output driver 114 are not present. Furthermore, the device 100 also includes one or more accelerated processing devices (APDs) 116. In at least some embodiments, the APDs 116 are coupled, for example, to one or more display devices 118. The APDs 116 may be part of the output device 110 or may be separate from the output device. The APDs 116 receive compute commands and graphics rendering commands from the processor 102, process these compute commands and graphics rendering commands, and further provide pixel outputs for display to the display device 118.

[0027] As will be described in more detail below, the APD116 includes one or more parallel processing units for performing computations according to the Single Instruction Multiple Data (SIMD) paradigm. In one or more embodiments, the APD116 is used to implement a GPU, in which case the parallel processing units are referred to as shader cores or streaming multiprocessors (SMX). Each parallel processing unit includes one or more processing elements, such as scalar and / or vector floating-point units and arithmetic logic units (ALUs). In various embodiments, the parallel processing units also include dedicated-purpose processing units (not shown), such as inverse square root units and sine / cosine units.

[0028] While this specification describes various functions performed by or in conjunction with the APD116, various alternatives exist, such as the functions described as being performed by the APD116 being additionally or alternatively performed by other computing devices having similar functions that are not driven by a host processor (e.g., processor 102) and provide a graphical output to a display device 118. For example, any processing system that performs processing tasks according to a SIMD paradigm is capable of performing the functions described herein. Alternatively, a computing system that does not perform processing tasks according to a SIMD paradigm is also capable of performing the functions described herein.

[0029] Figure 2 is a block diagram of device 100 showing further details related to the execution of processing tasks on APD116. In one or more embodiments, the processor 102 maintains one or more control logic modules in system memory 104 for the processor 102 to execute. The control logic modules include, for example, an operating system 202, a kernel-mode driver 204, and an application 206. These control logic modules control various functions of the operation of the processor 102 and APD116. For example, the operating system 202 communicates directly with the hardware and provides an interface to the hardware for other software running on the processor 102. The kernel-mode driver 204 controls the operation of APD116, for example, by providing an application programming interface (API) to software running on the processor 102 (e.g., application 206) to access various functions of APD116. In one or more embodiments, the kernel-mode driver 204 also includes a just-in-time compiler that compiles programs for execution by the processing components of APD116 (e.g., SIMD units 210).

[0030] In one or more embodiments, the APD116 includes any coordinated set of hardware and / or software that perform functions and calculations related to accelerating graphics processing tasks, data parallel tasks, nested data parallel tasks in an accelerated manner with respect to resources such as conventional CPUs, conventional GPUs, and combinations thereof. While GPU, Accelerated Processing Unit (APU), and General-Purpose Graphics Processing Unit (GPGPU) are commonly used terms in the art, the expression “Accelerated Processing Device (APD)” as used herein is a broader expression.

[0031] The APD116 executes commands and programs for selected functions, such as graphics calculations and non-graphics calculations suitable for parallel processing. In one or more embodiments, the APD116 is used to perform graphics pipeline calculations, such as pixel calculations, geometric calculations, and image rendering to the display device 118, based on commands received from the processor 102. The APD116 also performs calculations not directly related to graphics calculations, such as calculations related to video, physical simulations, computational fluid dynamics, etc., based on commands received from the processor 102. For example, such commands include special instructions not typically defined in the instruction set architecture (ISA) of the APD116. In at least some embodiments, the APD116 receives image geometry representing a graphics image along with one or more commands or instructions for rendering and displaying the image. In various embodiments, the image geometry corresponds to a representation of a two-dimensional (2D) or three-dimensional (3D) computer graphics image.

[0032] The APD116 includes a compute unit 208 (illustrated as compute units 208-1 to 208-3) such as a processing core, which includes one or more SIMD units 210 (illustrated as SIMD units 210-1 to 210-6) that perform operations in parallel according to the SIMD paradigm in response to a request from the processor 102. The SIMD paradigm is a system in which multiple processing elements can execute the same program, but with different data, by sharing a single program control flow unit and program counters. In one embodiment, each SIMD unit 210 includes a predetermined number of lanes, and each lane executes the same instruction simultaneously with other lanes in the SIMD unit 210, but with different data. Lanes can be switched off by prediction if it is not necessary for all lanes to execute a predetermined instruction. Prediction can also be used to execute a program with a branch control flow. More specifically, for programs with conditional branch instructions or other instructions where the control flow is based on calculations performed by individual lanes, the prediction of lanes corresponding to control flow paths not currently being executed and the serial execution of different control flow paths enable arbitrary control flows.

[0033] The basic unit of execution in compute unit 208 is a work item. Each work item represents a single instance of a program to be executed in parallel on a specific lane. Work items can be executed concurrently as a "wavefront" on a single SIMD unit 210. One or more wavefronts belong to a "workgroup," which is a collection of work items designated to run the same program. A workgroup can be executed by executing each of the wavefronts that make up the workgroup. Alternatively, wavefronts can be executed sequentially on a single SIMD unit 210, or partially or fully in parallel on different SIMD units 210. A wavefront is the largest collection of work items that can be executed concurrently on a single SIMD unit 210. Therefore, if a command received from processor 102 indicates that a particular program is too parallel to run concurrently on a single SIMD unit 210, the program is split into wavefronts that are parallelized on two or more SIMD units 210, or serialized on the same SIMD unit 210 (or both parallelization and serialization are performed as needed). The scheduler 212 performs operations related to scheduling various wavefronts on different compute units 208 and SIMD units 210.

[0034] The parallelism provided by one or more compute units 208 is suitable for graphics-related operations such as pixel value calculation, vertex transformation, tessellation, geometry shading operations, and other graphics operations. Therefore, in one or more embodiments, the graphics processing pipeline 214 receives graphics processing commands from the processor 102 and provides computational tasks for parallel execution to the compute units 208. Some graphics pipeline operations, such as pixel processing and other parallel computation operations, require the same command stream or compute kernel to be executed on a stream or collection of input data elements. Each instantiation of the same compute kernel is executed simultaneously on multiple SIMD 210 units within one or more compute units 208 to process such data elements in parallel. As referred to herein, for example, a compute kernel is a function containing instructions declared in a program and executed on the APD compute unit 208. This function is also referred to as a kernel, shader, shader program, or program.

[0035] The graphics processing pipeline 214 includes multiple stages (not shown in Figure 2 for brevity) configured to process various primitives simultaneously in response to draw calls. In one or more embodiments, the stages of the graphics processing pipeline 214 process various primitives generated by an application 206, such as a video game, simultaneously. When geometry data is sent to the graphics processing pipeline 214, a hardware state setting is selected to define the state of the graphics processing pipeline 128. Examples of states include the rasterizer state, blend state, depth stencil state, primitive topology type of the sent geometry, and shaders used to render the scene (e.g., vertex shader, domain shader, geometry shader, hull shader, pixel shader, etc.). Shaders implemented in the graphics pipeline state are represented by corresponding bytecode. In some cases, the information representing the graphics pipeline state is hashed or compressed to provide a more efficient representation of the graphics pipeline state.

[0036] As will be described in more detail below, the APD116 is configured to implement the features of this disclosure by performing several functions. For example, the APD116 is configured to implement a super resolution upscaler 332 that receives a low resolution rendered frame 502 of a video stream. The super resolution upscaler 332 spatially upscales the low resolution rendered frame 502 using temporal feedback (e.g., previously upscaled frame(s) of the video stream) to reconstruct a high resolution frame 508 representing the rendered frame.

[0037] Figure 3 shows further details of the graphics pipeline 214 shown in Figure 2, relating to one or more embodiments. As described below, the graphics pipeline 214 is capable of performing super-resolution upscaling using temporal feedback (e.g., data associated with one or more previously upscaled frames) to reconstruct high-resolution images while maintaining or improving image quality compared to native rendering. In at least some embodiments, the graphics pipeline 214 is implemented in the APD116 shown in Figures 1 and 2. It should be understood that the super-resolution upscaling techniques described herein can also be applied to other pipeline configurations.

[0038] In one or more embodiments, the graphics processing pipeline 214 is configured to render graphics as an image depicting a scene having three-dimensional geometry in a virtual space (also referred to herein as “world space”), and possibly having two-dimensional geometry. Typically, the graphics processing pipeline 214 receives a representation of a three-dimensional scene, processes that representation, and then outputs a two-dimensional raster image. These stages of the graphics processing pipeline 214 first process data that is the properties of the endpoints (or vertices) of geometric primitives, which provide information about the object to be rendered. Typical primitives in three-dimensional graphics include triangles and lines, and the vertices of these geometric primitives provide information such as xyz coordinates, texture, and reflectivity.

[0039] The graphics pipeline 214 has access to storage resources 334 (also referred to herein as “storage components”), such as one or more memory or cache hierarchies used to implement buffers and store vertex data, texture data, etc. The storage resources 334 are implemented, for example, using several embodiments of the system memory 104 shown in Figures 1 and 2. Some embodiments of the storage resources 334 include (or are accessible from) one or more caches 336, one or more random access memory units 338, video random access memory units (or more) (not shown), one or more processor registers (not shown), etc., depending on the nature of the data at a particular stage of rendering. Thus, it will be understood that the storage resources 334 refer to processor-accessible memory used by the graphics processing pipeline 214.

[0040] The graphics processing pipeline 214 includes stages, each performing a specific function. The stages represent a subdivision of the graphics processing pipeline 214's functions. Each stage is executed either partially or entirely as a shader program on a programmable processing unit, such as the SIMD unit 210 in Figure 2, or partially or entirely as non-programmable hardware with fixed functions located outside the programmable processing unit. Stages 301 and 303 represent the front-end geometry processing of the graphics processing pipeline 214 before rasterization. Stages 305 through 311 represent the back-end pixel processing of the graphics processing pipeline 214.

[0041] In the input assembler stage 301 of the graphics pipeline 214, the input assembler 302 is configured to access information from a storage resource 334 used to define objects that represent parts of a scene model. For example, in various embodiments, the input assembler stage 220 reads primitive data (e.g., points, lines, and / or triangles) from a user-submitted buffer (e.g., a buffer submitted at the request of software running on the processor 102, such as application 206) and assembles the data into primitives used by other pipeline stages of the graphics processing pipeline 200. As used herein, the term “user” refers to application 206 or other entity that provides shader code and three-dimensional objects for rendering to the graphics processing pipeline 214. The term “user” is used to distinguish the entire activity performed by APD 116. The input assembler 302 assembles the vertices into several different primitive types (such as line lists, triangle strips, or adjacent primitives) based on the primitive data contained in the buffer submitted by the user, and formats the assembled primitives for use by the rest of the graphics processing pipeline 214.

[0042] In one or more embodiments, the graphics processing pipeline 214 operates on one or more virtual objects having geometry defined relative to coordinates in a scene, defined by a set of vertices set in world space. For example, the input data used by the graphics processing pipeline 214 includes a polygon mesh model of scene geometry whose vertices correspond to primitives processed in the rendering pipeline according to an aspect of this disclosure, and this initial vertex geometry is set in a storage resource 334 during an application stage executed by the CPU.

[0043] In the vertex processing stage 303 of the graphics pipeline 214, one or more vertex shaders 304 process the vertices of a primitive using an input assembler 302. For example, a vertex shader 304 receives a single vertex of a primitive as input and outputs a single vertex. The vertex shader 304 performs various per-vertex operations such as transformations, skinning, morphing, and per-vertex lighting. Transformation operations include various operations for transforming the coordinates of a vertex (e.g., XY coordinates and Z depth value). These operations include one or more of modeling transformations, view transformations, projection transformations, perspective splitting, and viewport transformations. In this specification, such transformations are considered to modify the coordinates or "position" of the vertex being transformed. Other operations of the vertex shader 304 modify attributes other than coordinates.

[0044] In one or more embodiments, the vertex shader(s) 304 is implemented partially or entirely as a vertex shader program that runs on one or more compute units 208. The vertex shader program is provided by the processor 102 and is based on a program pre-written by a computer programmer. The kernel-mode driver 204 compiles such computer programs to produce a vertex shader program in a format suitable for execution within the compute units 208. Some embodiments of shaders, such as the vertex shader 304, implement large-scale single-instruction multiple-data (SIMD) processing so that multiple vertices are processed simultaneously. In at least some embodiments, the graphics pipeline 214 implements a unified shader model so that all shaders included in the graphics pipeline 214 have the same execution platform on a shared large-capacity SIMD compute unit 210. In such embodiments, the shaders, including the vertex shader(s) 304, are implemented using a common set of resources referred to herein as the unified shader pool 306.

[0045] In one or more embodiments, the vertex processing stage 303 performs additional vertex processing calculations to subdivide the primitive and generate new vertices and new geometry in world space. In at least some embodiments, these additional vertex processing calculations are performed by one or more of the hull shader 308, tessellator 310, domain shader 312, and geometry shader 314. The hull shader 308 operates the input higher-order patch or control points used to define the input patch. The hull shader 308 outputs tessellation coefficients and other patch data. In one or more embodiments, the primitive generated by the hull shader 308 is provided to the tessellator 310. The tessellator 310 receives the object (patch, etc.) from the hull shader 308 and generates information identifying the primitive corresponding to the input object by tessellating the input object based, for example, the tessellation coefficients provided to the tessellator 310 by the hull shader 308. Tessellation divides input higher-order primitives, such as patches, into a set of lower-order output primitives that represent higher-granularity details, for example, indicated by a tessellation coefficient that specifies the granularity of the primitives generated by the tessellation process. Thus, a scene model is represented by fewer higher-order primitives (to save memory or bandwidth), and additional details are added by tessellating the higher-order primitives.

[0046] Domain shader 312 takes the domain's location and (if necessary) other patch data as input. Domain shader 312 operates based on the provided information and generates a single vertex for output based on the input domain location and other information. Geometry shader 314 receives an input primitive and outputs up to four primitives generated by geometry shader 314 based on the input primitive. In some embodiments, geometry shader 314 retrieves vertex data from storage resource 334 and generates new graphics primitives such as lines and triangles from the vertex data of storage resource 334. Specifically, shader 314 retrieves the vertex data of an entire primitive and generates zero or more primitives. For example, geometry shader 314 can operate on a triangle primitive with three vertices. Geometry shader 314 can perform a variety of different types of operations, including point sprint expansion, dynamic particle system operation, fur fin generation, shadow volume generation, single-pass render to cubemap, per-primitive material swapping, and per-primitive material setup. In at least some embodiments, one or more of the hull shader 308, domain shader 312, or geometry shader 314 are implemented as shader programs executed on a programmable processing unit such as a SIMD unit 210, while the tessellator 310 is implemented by fixed-function hardware.

[0047] Once the front-end processing is complete, the scene is defined by a set of vertices, each having a set of vertex parameter values ​​stored in the storage resource 334. In a particular embodiment, the vertex parameter values ​​output from the vertex processing stage 303 include positions defined in different homogeneous coordinates for different zones.

[0048] As described above, stages 305 to 311 represent the backend processing of the graphics processing pipeline 214. The rasterizer stage 305 includes a rasterizer 316 that accepts and rasterizes simple primitives and the generated upstream. The rasterizer 316 performs operations such as shading, clipping, oblique splitting, scissoring, and viewport selection. The rasterizer 316 generates a set of pixels that will subsequently be processed in the pixel processing / shader stage 307 of the graphics pipeline 214. In some embodiments, the set of pixels includes one or more tiles. In one or more embodiments, the rasterizer 316 is implemented by fixed-function hardware.

[0049] The pixel processing stage 307 includes one or more pixel shaders 318 that take a pixel flow (including, for example, a set of pixels generated by the rasterizer 316) as input and output zero or more pixel flows depending on the input pixel flow. The pixel shaders 318 calculate the output values ​​of screen pixels based on the primitives and rasterization results generated upstream. In one or more embodiments, the pixel shaders 318 apply textures from texture memory, which can be implemented as part of the storage resource 334. Each of the outputs(s) generated by the one or more pixel shaders 318, such as color values, depth values, and stencil values, is stored in one or more corresponding buffers, such as a color buffer 320, a depth buffer 322, and a stencil buffer 324. The combination of the color buffer 320, the depth buffer 322, and optionally the stencil buffer 324 is referred to as the frame buffer 326. In at least some embodiments, the graphics pipeline 214 implements multiple frame buffers 326, including intermediate buffers such as a front buffer, a back buffer, a render target, and a frame buffer object. The calculations of the pixel shader 318 are performed by a shader program that runs on the programmable processing unit 210.

[0050] In one or more embodiments, a pixel shader 318 or another shader accesses shader data, such as texture data, stored in a storage resource 334. The texture data defines a texture, which is a bitmap image used at various points in the graphics processing pipeline 214. For example, in some cases, the pixel shader 318 applies a texture to pixels to improve the apparent rendering complexity without increasing the number of vertices being rendered (e.g., to provide a more "photographic" appearance). In other cases, the vertex shader 304 uses the texture data to modify primitives, increasing complexity by, for example, generating or modifying vertices to improve aesthetics. For example, the vertex shader stage 304 modifies the displacement of vertices using a height map stored in the storage resource 334. This type of technique can be used, for example, to produce water with a more realistic appearance compared to using only a texture in the pixel processing stage 307, by changing the position and number of vertices used when rendering water. In some cases, the geometry shader 314 accesses texture data from the storage resource 334.

[0051] The output merger stage 309 receives the output from the pixel processing stage 307, merges those outputs, performs operations such as Z-testing, alpha blending, and stenciling on each pixel received from the pixel shader 318, and includes an output merger 328 that determines the final color of the screen pixels. For example, the output merger 328 combines various types of output data (e.g., pixel shader values, depth, and stencil information) with the contents of the color buffer 320, depth buffer 322, and optionally stencil buffer 324, and stores the combined output back in the frame buffer 326. The output 309 of the merger stage can be referred to as rendered pixels that collectively form a rendered frame. In one or more embodiments, the output merger 328 is implemented by fixed-function hardware.

[0052] The post-processing stage 311 is executed after the output merger stage 309. In the post-processing stage 311, one or more post-processors 330 operate on the rendered frame stored in the frame buffer 326 (or on each pixel) to apply one or more post-processing effects, such as ambient occlusion or tone mapping, before the frame is output to the display. In at least some embodiments, the post-processors 330 are implemented using one or more vertex shaders 304, one or more pixel shaders 318, etc. The post-processed frame is written to the frame buffer 326, such as a back buffer for display or an intermediate buffer for further post-processing. In at least some embodiments, the graphics pipeline 214 includes other shaders or components, such as a computer shader 340, a ray tracer 342, a mesh shader 344, etc., and can communicate with one or more other components of the graphics pipeline 214. At least a portion of the vertex processing stage 303, the rasterizer stage 305, the pixel processing stage 307, and the post-processing stage 311 are collectively referred to herein as the "renderer 313" or "rendering stage 313" of the graphics pipeline 214.

[0053] In many cases, the amount of processing resources required to render a full high-resolution image can make it difficult to render frames while meeting the current frame rate, such as at least 60 frames per second (fps). Therefore, in one or more embodiments, the application 206 (e.g., a video game) uses the renderer 313 to generate images rendered at a lower render resolution or size (e.g., 1920 x 1080 pixels) than one or more final output / presentation resolutions or sizes (e.g., 3840 x 2160 pixels) in order to meet timing requirements and reduce processing resource requirements. In at least some embodiments, a super-resolution upscaling stage 315, including a super-resolution upscaler 332 (hereinafter referred to as "upscaler 332" for brevity), processes the low-resolution rendered image to generate an upscaled image that represents the content of the low-resolution rendered image at a resolution equal to (or at least close to) the target presentation resolution. In one or more embodiments, the upscaling stage 315 is performed as part of a post-processing stage 311. However, in other embodiments, the upscaler 332 is part of a different processing stage in the graphics pipeline 214. In one or more embodiments, the upscaler 332 is implemented using one or more vertex shaders 304, pixel shaders 318, compute shaders 340, etc., or a combination thereof.

[0054] The upscaler 332 improves the rendering performance of the application by implementing a temporal upscaling algorithm that operates on a large number of inputs. Therefore, in one or more embodiments, the upscaler 332 is positioned in the graphics pipeline 214 to ensure the best balance between visual quality and performance. Placing other image space processes before the upscaler has the advantage of providing a performance benefit to the application 206, as these other image space processes run at a lower resolution. However, depending on the type of image space process / technology, this placement may not be appropriate. For example, some image space processing may introduce noise or grain into the final image (e.g., to simulate a physical camera). Placing such image space processing before the upscaler may cause the upscaler to amplify the noise, resulting in undesirable artifacts in the final upscaled image. Therefore, in one or more embodiments, the upscaler 332 is typically placed between a post-processor 330 that operates on the current frame at the render resolution and a post-processor 330 that operates on the upscaled frame generated by the upscaler 332 at the presentation resolution. However, in at least some embodiments, techniques such as noise and grain can be calculated at low resolution and applied / combined to an upscaled frame, for example, using bilinear sampling.

[0055] Figure 4 shows an integrated graphics pipeline diagram 400 for the upscaler 332. In this embodiment, the first stage is the rendering section 313 of the graphics pipeline 214, which renders the current frame at a predetermined render resolution in the render color space. The pre-upscaling post-processing stage 311-1 receives the rendered frame as input and performs one or more post-processing operations on the rendered frame at the render resolution and in the render color space. The pre-upscaling post-processing stage 311-1 typically includes a post-processor 330 that performs post-processing operations that access the depth buffer 322 of the rendered frame. Examples of such post-processing operations include screen-space reflections, screen-space ambient occlusion, denoising (e.g., shadows or reflections), exposure, etc. The upscaling stage 315 receives the output of the pre-upscaling post-processing stage 311-1 (or the rendering stage 313) as input and performs at least one of upscaling or anti-aliasing operations on the pre-scaled post-processed frame or the rendered frame (if no post-processing has been performed) in the linear color space. The output of the upscaling stage 315 is at least one of an upscaled or anti-aliased frame having a target presentation resolution in the presentation color space. The post-upscaling post-processing stage 311-2 takes the upscaled (and anti-aliased) frame as input and performs one or more post-processing operations on the upscaled frame at the presentation resolution and in the presentation color space. The post-upscaling post-processing stage 311-2 typically includes a post-processor 330 that either uses the anti-aliased frame or performs operations that add effects to the frame that would result in undesirable artifacts when the frame is upscaled. Examples of such post-processing operations include film grain, chromatic aberration, vignette, tone mapping, blooming, depth of field, motion blur, etc.In the embodiment shown in Figure 4, the final stage 417 is the presentation of a user interface or heads-up display, which is rendered by the graphics pipeline 214 at the presentation resolution and within the presentation color space.

[0056] Figure 5 is a block diagram showing a more detailed view of the upscaler 332. As described below, the upscaler 332 is temporal and takes as input data associated with the currently rendered (aliased) frame 502 (also referred to herein as the “current frame 502” or “input frame 502”) and data associated with previously presented (upscaled) frames. In one or more embodiments, the rendered frame 502 and the previously upscaled frame are frames of a video stream. The previously presented frame is the most recent frame processed by the upscaler 332 and presented to the user at the target presentation resolution. In one or more embodiments, the input data associated with the previously presented frame includes an output buffer 506-1 that holds the data, such as a color buffer (e.g., output color), a depth buffer, etc., for the previously presented frame. The resolution / size of the output buffer 506-1 corresponds to the presentation resolution.

[0057] In one or more embodiments, the input data associated with the current frame 502 includes various buffers provided by Application 206, such as a color buffer 320-1, a depth buffer 322-1, and a motion vector buffer 504. The input color buffer 320-1 is a render-resolution color buffer for the current frame 502 provided by Application 206. In one or more embodiments, the input color buffer 320-1 includes color data such as pixel color values ​​generated based on subpixel jittering performed during rendering of the current frame 502. Subpixel jittering can be performed for each frame in a sequence of frames during the rendering stage. During subpixel jittering, the center point for color determination of a pixel in a frame is slightly shifted to another point within the pixel using a subpixel offset, and color data is determined for this offset position. In one or more embodiments, the output merger 328 of the graphics pipeline 214 combines the multisampled color data determined for each subpixel sample to determine the final color data for the corresponding pixel. Next, the output merger 328 stores the final color data in the color buffer 320-1 of frame 502. In one or more embodiments, the jitter locations are determined randomly or according to a predetermined pattern or sequence such as a Halton sequence (e.g., Halton[2,3]). The Halton sequence provides spatially isolated points that cover the available pixel space. In one or more embodiments, jitter is applied to the rendering of several object types, including opaque objects, alpha-transparent objects, and ray-traced objects. For rasterized objects, subpixel jittering values ​​can be applied to a camera projection matrix, which is then used to perform transformations during vertex shading. For ray-traced renderings, subpixel jitter is applied to the ray origin, which is usually the camera position.

[0058] In some configurations, the input depth buffer 322-1 includes the stencil buffer 324. However, in other configurations, the input depth buffer 322-1 is stored separately from the stencil buffer 324. The input depth buffer 322-1 is the render resolution depth buffer for the current frame 502 provided by application 206. In one or more embodiments, the resolution / size of the input color buffer 320-1 and the input depth buffer 322-1 are equal to the render resolution. The motion vector buffer 504 includes a two-dimensional motion vector that encodes the motion from a pixel in the current frame 502 to the same pixel position in a previous (upscaled) frame. In one or more embodiments, the motion vector is provided by application 206, for example, in the range [(<-width,-height>...<-width,height>)]. For example, the value for the pixel in the upper left corner of the screen is...<width,height> The motion vector represents movement traversing the entire width and height of the input surface, starting from the lower right corner of the screen. In at least some configurations, one or more opaque objects, alpha-tested objects, or alpha-blended objects write their motion vectors for all covered pixels. If vertex shader effects such as UV scrolling are applied, these calculations are added to the motion calculation, resulting in an enhanced outcome. In some configurations, the resolution / size of the motion vector buffer 504 is equal to the render resolution. However, in other configurations, the resolution / size of the motion vector buffer 504 is equal to the presentation resolution. In one or more embodiments, the color buffer 320-1, depth buffer 322-1, and motion vector buffer 504 are texture data types, but other data types are similarly applicable.

[0059] In at least some embodiments, when upscaling the rendered frame 502, the upscaler 332 processes additional external input resources such as the reactive mask 602 (Figure 6) and exposure 604 (Figure 6) of the rendered frame 502. In the context of the upscaler 332, "reactive" means the extent to which the rendered samples for the current frame 502 have influence on the generation of the final upscaled image. In one or more embodiments, the rendered samples for the current frame 502 contribute, with some exceptions, to the result calculated by the upscaler 332. For example, to produce the best results for fast-moving alpha-blended objects, one or more stages of the upscaler 332 (e.g., the reprojection accumulation stage 611 (Figure 6)) can be configured to be more reactive to such pixels. Since it can be difficult to determine from color, depth, or motion vectors which pixels were rendered using alpha blending, in one or more embodiments, application 206 provides a reactivity mask 602 as input to the upscaler 332. The reactivity mask 602 provides a mechanism for application 206 to identify areas of the rendered frame 502 that do not leave a footprint in the input depth buffer 322-1 or that contain motion vectors 504. In other words, the reactivity mask 602 reduces the upscaler 332's reliance on past information when compositing current pixels and instead guides where samples of the current frame 502 should contribute more to the final result. The reactivity mask 602 allows application 206 to provide values ​​from, for example, [0...1], where a value of 0 indicates that the pixel is not reactive at all (e.g., the upscaler 332 uses its default compositing strategy), and a value of 1 indicates that the pixel is fully reactive. In one or more embodiments, the alpha value used when compositing an alpha-blended object (e.g., particles) into a scene is implemented as a surrogate for reactivity.In these embodiments, application 206 writes the alpha value of each pixel covered for the alpha-blended object to the corresponding pixel in the reactive mask 602. In one or more embodiments, the resolution / size of the reactive mask 602 is equal to the render resolution. Furthermore, in one or more embodiments, the reactive mask 602 is a texture data type, but other data types can be applied in the same way.

[0060] Exposure 604 informs the upscaler 332 of the exposure value calculated by application 206 for the rendered frame 502. In one or more embodiments, the exposure value matches the exposure used by application 206 during the next tone mapping pass. In at least some embodiments, the upscaler 332 calculates its own exposure value for internal use at various stages. Furthermore, in one or more embodiments, the output generated by the upscaler 332 has this internal tone mapping inverted before the final output is written. That is, the upscaler 332 returns a result in the same domain (or a domain close to it) as the original input signal. In one or more embodiments, exposure 604 is a texture data type, but other data types can be applied as well.

[0061] The upscaler 332 uses the current frame input and previous frame inputs to generate a super-resolution upscaled (and anti-aliased) frame 508 (hereinafter also referred to herein as the “upscaled frame 508” for brevity) corresponding to the rendered frame 502 at the target presentation resolution. In one or more embodiments, the upscaler 332 stores data representing the upscaled frame 508 in an output buffer 506-2. The upscaler 332 maintains an output buffer history 510 for accessing output buffers 506-1 generated for previously presented frames. The upscaler 332 implements one or more components for performing the upscaling and anti-aliasing operations described herein. For example, the upscaler 332 implements an auto-exposure component 512, an input color adjustment component 514, a reconstruction extension component 516, a depth clipping component 518, a locking component 520, a reprojection accumulation component 522, and a sharpening component 524. As will be discussed later with respect to Figure 6, each of these components is implemented in the corresponding stage of the upscaler 332. Although components 512-524 of the upscaler 332 are shown as being implemented separately from each other, it should be understood that two or more of these components may be combined.

[0062] Figure 6 illustrates the various stages / paths of the upscaler 332, including the automatic exposure stage 601, the input color adjustment stage 603, the reconstruction extension stage 605, the depth clipping stage 607, the locking stage 609, the reprojection accumulation stage 611, and the sharpening stage 613. Figure 6 further illustrates the inputs processed by each stage and the outputs generated by each stage. In the embodiments shown in Figure 6, the shaded frames represent the stages of the upscaler 332, the dashed frames represent the input / output buffers, and the solid frames represent the intermediate / processing buffers. In one or more embodiments, application 206 interacts with the upscaler 332 through one of several different application programming interfaces (APIs). For example, application 206 can instantiate the upscaler 332, make one or more calls to the upscaler 332, or pass one or more data structures and inputs to the upscaler 332 through the API. In one or more embodiments, when the upscaler 332 is instantiated (for example, by application 206), storage resources 334, such as GPU local memory, are allocated for consumption by the upscaler 332. The upscaler 332 uses these storage resources 334 to store intermediate textures computed by the upscaler 332, and further to store persistent textures across many frames of application 206.

[0063] In the automatic exposure stage 601, application 206 provides the color buffer 320-1 of the currently rendered frame 502 as input to the automatic exposure component 512. If the content of the color buffer 320-1 is high dynamic range (HDR), application 206 can indicate this to the automatic exposure component 512, for example, by setting a flag in a data structure provided by application 206 to the upscaler 332. The automatic exposure component 512 processes the color buffer 320-1 to generate up to two intermediate storage resources, depending on the configuration provided by application 206. The first intermediate storage resource is the current luminance texture 606, which represents the luminance of the input color at a low resolution. For example, the current luminance texture 606 is a texture that is 50% (or another percentage) of the render resolution texture (e.g., color buffer 320-1) containing the luminance values ​​of the currently rendered frame 502. As described below, the current luminance texture 606 is used by the shading change detection process in the reprojection accumulation stage 611. The second intermediate storage resource is an exposure texture 604 containing exposure values ​​calculated for the rendered frame 502. The exposure texture 604 is optionally used in subsequent stages of the upscaler 332, depending on the configuration of the upscaler 332. For example, application 206 may indicate whether the exposure 604 should be used during the upscaling process by setting a flag in a data structure provided to the upscaler 332, for example. In other embodiments, the exposure 604 is provided to the automatic exposure component 512 as input by application 206. Where appropriate, the exposure 604 is used by the exposure calculation of the input color adjustment stage 603 to apply tone mapping, and further used by the reprojection accumulating stage 611 to invert the local tone mapping before the upscaler 332 generates the output (e.g., the upscaled frame 508). In one or more embodiments, the resolution / size of the exposure 604 is 1x1 pixels, but other sizes are equally applicable.

[0064] In at least some embodiments, the auto-exposure component 512 implements at least one downsampling mechanism, such as an AMD FidelityFX® Single Pass Downsampler, and generates a mipmap chain using a shader dispatch, such as a single compute shader dispatch. A mipmap is a collection of successively reduced-resolution bitmap images of a texture. In other words, a mipmap contains multiple versions of the same texture, each version having a different resolution. These different versions are sometimes called "mipmap levels," "levels," or "mips." In one or more embodiments, unlike the conventional pyramidal approach, the downsampling mechanism implemented by the auto-exposure component 512 not only generates a specific set of mipmap levels for any input texture (e.g., input color buffer 320-1), but also performs some calculations on that data as the data is stored at a target location in memory.

[0065] In one or more embodiments, the downsampling mechanism of the auto-exposure component 512 is configured to write only to a second (e.g., half-resolution) mipmap level 702 and a final (1x1) mipmap level 704, as shown in Figure 7. Furthermore, different calculations are applied to each of these levels 702 and 704 to calculate the amount to be used in the next stage of the upscaler 332. Thus, the remaining levels of the mipmap chain do not need to be supported by GPU local memory (or other types of memory). The second mipmap level 702 contains the current luminance 606, and the final mipmap level 704 contains the exposure 604, and these values ​​are calculated during the downsampling of the color buffer 320-1. In one embodiment, the current luminance 606 contains a texture with a render resolution of 1 / 32 such that each pixel contains the average luminance of a 32x32 pixel area in the source texture. By sampling the texture using the coordinates of the current pixel, and then using bilinear sampling, the average brightness of a 32x32 pixel area centered on the input pixel is obtained.

[0066] Referring to Figure 6, in one or more embodiments, the various stages of the upscaler 332 operate in a color space such as YCoCg that is different from the rendered frame 502 (e.g., RGBa). Therefore, to avoid repeatedly calculating conversions from the color space used by application 206, the upscaler 332 implements an input color adjustment stage 603 that applies all adjustments to the color in one step. For example, in the input color adjustment stage 603, the input color adjustment component 514 takes as input the input color buffer 320-1 provided by application 206 and the exposure 604 provided by application 206 or generated by the automatic exposure component 512 of the automatic exposure stage 601. The input color adjustment component 514 processes these inputs and performs various adjustment operations on the input color of the rendered frame 502. One example of such adjustment operations is to divide the input color (e.g., RGB) by the pre-exposure value provided by application 206 as part of the current frame 502 to obtain the original input color. In many cases, different frames use different pre-exposure values ​​(to aid computational accuracy from a game perspective), and further, all images are placed in the same range space by dividing the original input color of the frame by the pre-exposure value. Another example of the adjustment operation involves aligning the input image to a mid-gray level by multiplying the input color by the exposure value 604. A further example of the adjustment operation involves converting the post-exposure color to the YCoCg color space. The YCoCg color space includes a lumen value (Y) and two chroma values, chrominance green (Cg) and chrominance orange (Co). The lumen value (Y) represents the luminance in the rendered frame 502 (the achromatic portion of the frame), and the chroma components (Co and Cg) represent the color information of the rendered frame 502. The result of the input color adjustment stage 603 is then cached in an adjusted color buffer (texture) 608, which can be read in subsequent stages of the upscaler 332. In one or more embodiments, the resolution / size of the adjusted color buffer 608 is equal to the render resolution.

[0067] As part of the adjustment process, the input color adjustment component 514 generates a luminance history buffer (texture) 610 whose resolution / size is equal to the render resolution. The luminance history buffer 610 contains the lumens Y of the currently prepared input color for the rendered frame 502. In one or more embodiments, the luminance history buffer 610 is persistent (i.e., not available for aliasing or cleared per frame). Thus, multiple frames of luminance history (e.g., 4 frames) are retained in the luminance history buffer 610 and are accessible during the input color adjustment stage 603 for any one frame. However, at the end of the input color adjustment stage 603, the luminance history values ​​are shifted downwards. That is, in at least some embodiments, the next stage of the upscaler 332 can access a smaller number of luminance frames, e.g., the three most recent frames of luminance (the current frame and two previous frames). Thus, in this embodiment, if the current frame is n, the values ​​stored in the luminance history buffer 610 are as follows:

[0068] [Table 1]

[0069] In one or more embodiments, the input color adjustment component 514 encodes a stability coefficient in the alpha channel of the luminance history buffer 610. This stability coefficient is a measurement of luminance stability over the current frame 502 and a predetermined number of frames (e.g., 3 frames) that preceded the current frame 502.

[0070] In addition to performing the input color adjustment operations described above, the input color adjustment component 514 clears the reprojected / previous depth buffer 616 to a known value, as described later. This process prepares the previous depth buffer 616 for the reconstruction extension stage 605 of the next rendered frame of application 206. The input color adjustment component 514 selects a clear value based, for example, on the configuration of the previous depth buffer 616. For example, the input color adjustment component 514 clears the previous depth buffer 616 to a maximum z-far value, which is typically 0 for the inverted depth. In at least some configurations, the previous depth buffer 616 is cleared as a result of the reconstruction extension stage 605 feeding into the previous depth buffer 616 using an integral operation.

[0071] In the reconstruction extension stage 605, the reconstruction extension component 516 receives the input depth buffer 322-1 and motion vector buffer 504 provided by the application 206 as input. The reconstruction extension component 516 processes this input to generate an extended depth buffer 612 for the previous frame, a buffer 614 containing an extended set of motion vectors in UV space, and a reprojected / previous depth buffer 616. In one or more embodiments, the reconstruction extension component 516 applies motion vector scaling to convert non-screen space motion vectors to screen space motion vectors before processing the motion vectors. The extended depth buffer 612 is a texture containing extended depth values ​​determined from the input depth buffer 322-1. The extended motion vector buffer 614 is a texture containing extended two-dimensional motion vectors determined from the input motion vector buffer 504. In one or more embodiments, the first color channel, such as the red channel, and the second color channel, such as the green channel, of the expanded motion vector buffer 614 contain two-dimensional motion vectors in normalized device coordinate (NDC) space. The previous depth buffer 616 is a texture containing the depth values ​​of the reconstructed previous frame. Each of these buffers has a resolution / size equal to the render resolution of the current frame 502.

[0072] More specifically, the reconstruction extension component 516 calculates the extended depth values ​​and motion vectors from the input depth values ​​and motion vectors contained in the input depth buffer 322-1 and input motion vector buffer 504, respectively, of the current frame 502. The extended depth values ​​and motion vectors highlight the edges of the geometry rendered in the input depth buffer 322-1. For example, the edges of geometry often introduce discontinuities into a continuous series of depth values. Therefore, when the depth values ​​and motion vectors are extended, they naturally follow the contours of the geometric edges present in the input depth buffer 322-1. In one or more embodiments, the reconstruction extension component 516 calculates the extended depth values ​​and motion vectors by considering the depth values ​​in a 3x3 (or other kernel size) neighborhood of each pixel in the current frame 502. Next, the reconstruction extension component 516 selects the nearest neighbor depth value and corresponding motion vector to the camera. This process is illustrated in Figure 8, which shows one embodiment of the geometry 802 of the current frame 502, in which the central pixel 804 of the 3x3 kernel 806 is updated with the depth value and motion vector from the pixel 808 with the maximum depth value. The reconstructed extension component 516 stores the determined extended depth value in the extended depth buffer 612 and the determined extended motion vector in the extended motion vector buffer 614.

[0073] The reconstructed extension component 516 uses the expanded motion vectors of the expanded motion vector buffer 614 to estimate the position of each pixel from the current frame's depth buffer 322-1 in the previous frame. For example, the reconstructed extension component 516 applies the expanded motion vectors calculated for the pixel to the values ​​in the expanded depth buffer 612 to determine the pixel's position in the previous frame. In other words, each depth sample from the expanded depth buffer 612 is reprojected to its position in the previous frame using the sample's expanded motion vector. The reprojected depth samples are then distributed among the affected depth samples, for example, using backprojection or inverse reprojection. Since many pixels may be reprojected to the same pixel in the previous frame, the reconstructed extension component 516 uses indivisible operation to resolve the value of the nearest depth value for each pixel. For example, in one or more embodiments, the reconstructed extension component 516 uses indivisible operation such as InterlockedMax or InterlockedMin provided by the Higher Level Shader Language (HLSL) or its equivalent. In some embodiments, the reconfiguration extension component 516 performs different integral operations (e.g., InterlockedMax or InterlockedMin) depending on whether the input depth buffer 322-1 is inverted or not. The reconfiguration extension component 516 stores the reconfigured / estimated depth value in the reconfigured / previous depth buffer 616.

[0074] Referring to Figure 6, in the depth clipping stage 607, the depth clipping component 518 receives the expanded depth buffer 612, the expanded motion vector buffer 614, and the previous depth buffer 616 as input. The depth clipping component 518 processes this input to generate a deocclusion mask / map 618 that shows the deocclusion areas of the currently rendered frame 502. For example, when the camera moves from an initial position (previous frame) to a new position (current frame), pixels that were initially occluded from the viewpoint of the camera's previous position may become visible (deoccluded) from the viewpoint of the camera's current position. In one or more embodiments, the deocclusion mask 618 is a texture containing values ​​that indicate how much the corresponding pixels in the current frame 502 are deoccluded. In one embodiment, a value of 0 indicates that the entire pixel was occluded in the previous frame and is now deoccluded, while a value of 1 indicates that the pixel was fully visible in the previous frame and is also fully visible in the current frame 502. A value between 0 and 1 indicates that the pixel was visible in the previous frame to a degree proportional to that value. In one or more embodiments, the de-occlusion mask 618 has the same resolution / size as the render resolution.

[0075] The depth clipping component 518 generates an occlusion mask 618 by, for example, calculating a pixel-by-pixel depth value from the previous camera position and the new camera position. The depth clipping component 518 compares the delta between depth values ​​to a separation value such as the Acheley separation constant ksep. This Acheley separation constant provides the minimum distance between two objects represented in a floating-point depth buffer, which the depth clipping component 518 uses to determine with high certainty that the pixels were originally distinct from each other. In one or more embodiments, if the depth clipping component 518 determines that the delta between depth values ​​is greater than the separation value calculated for the configuration of the application's depth buffer 612, the depth clipping component 518 determines that the pixels represent separate objects. However, if the depth clipping component 518 determines that the delta between depth values ​​does not exceed the separation value, the depth clipping component 518 cannot determine with certainty that the pixels represent separate objects. The depth clipping component 518 stores values ​​in the de-occlusion mask 618 for pixels in the range [0...1], for example, where a value of 1 is mapped to a delta greater than or equal to the isolation value.

[0076] In the locking stage 609 of the upscaler 332, the locking component 520 generates a new pixel lock based on the pixels of the current frame 502. As described below, the pixel lock is consumed in the reprojection accumulation stage 611 of the upscaler 332. The locking component 520 receives input lock state information 620, such as an input lock state texture / buffer 620-1 (also referred to herein for brevity as "lock state 620-1"), and the adjusted color buffer 608 as input. Based on this input, the locking component 520 generates a lock for one or more pixels of the current frame 502, where applicable. In one or more embodiments, the generated lock is stored in an output lock state texture / buffer 620-2 (also referred to herein for brevity as "lock state 620-2"). In one or more embodiments, the upscaler 332 maintains the lock state 620 as an array of two textures, namely input lock state 620-1 and output lock state 620-2. The input lock state 620-1 is a lock state texture generated for a previously upscaled frame and used as input by the upscaler 332 when processing the current frame 502. The output lock state 620-2 is a lock state texture generated for the current frame 502 and used as input for the next frame to be processed by the upscaler 332. Both the input lock state 620-1 and the output lock state 620-2 have the same resolution / size as the target (upscaled) presentation resolution and maintain the lock state 620. The two-texture array configuration of the lock state 620 allows neighboring pixels of a pixel being processed by the locking stage 609 to be read while avoiding read-modify-write contention. It should be understood that the locking stage 609 is not limited to image upscaling. For example, the locking stage 609 can be applied to temporal anti-aliasing (TAA) and other techniques.

[0077] In at least some embodiments, the input lock state 620-1 and the output lock state 620-2 each include lock state information indicating whether a pixel in the relevant frame is associated with a lock or unlocked. In one or more embodiments, a lock functions as a classifier indicating that a locked pixel contains high-frequency information (e.g., luma changes between opposing neighbors of a pixel). In at least some embodiments, a lock functions as a mask indicating that color correction in the reprojection accumulation stage 611 is not performed on the locked pixel. The net effect of a pixel being locked is that more color data from previous frames is used when calculating the final super-resolution pixel color during the reprojection accumulation stage 611. For example, if shading changes between frames, the subsequent color correction process may limit the influence of the history color so that the history color is closer to the color around the sample in the current frame 502, thereby clamping small / faint features that are not visible in all jittered renders and causing them to disappear from the blended image. Locking increases the contribution of samples containing small / faint features, preventing these features from being clamped and disappearing from the blended image.

[0078] In one or more embodiments, the lock state 620 consists of two values: a red channel value and a green channel value. In other embodiments, the lock state 620 consists of three values: a red channel, a green channel, and a blue channel. The red channel of the lock state 620 contains the remaining lifetime of the pixel lock and is initialized based on the jitter sequence length. For example, when performing a 2x upscaling, the sequence length is 32, and the lock may be active for 32 frames. The length of 32 frames is stored in the red channel as the initial lifetime of the lock and decreases by approximately 1 / 32 times (in the 2x upscaling example) with each frame. In one or more other embodiments, when a pixel is locked, the locking component 520 populates the red channel of the output lock state 620-2 with a remaining lifetime value indicating that the lock is new (i.e., created for the current frame) and whether the lock is the first / initial lock for that pixel or the next lock (e.g., the second lock). For example, the locking component 520 populates the red channel of the output lock state 620-2 with a remaining lifetime value of -1 to indicate that the lock is the first lock on the pixel, where the negative sign indicates that the lock is a new lock. If the pixel was locked in a previous frame, the new lock is considered the next pixel lock (e.g., the second lock), and the locking component 520 populates the red channel of the output lock state 620-2 with a remaining lifetime value of -2 to indicate that the lock is new (i.e., generated for the current frame) and a subsequent lock on the pixel. In one or more embodiments, if the initial value of the remaining lifetime of a lock is -1, the remaining lifetime is decremented between 0 and 1, and if the initial value of the remaining lifetime of a lock is -2, the remaining lifetime is decremented between 0 and 2. In these embodiments, rather than decrementing the remaining lifetime by 1 / 32 (in the example of 2x upscaling), the remaining lifetime is dynamically decremented according to the contribution of the locked pixel to the final color of the current frame. Furthermore, when the final lock of a pixel is stored in the reprojection accumulation stage 611, the absolute value of the remaining lifetime is also stored.Therefore, in at least some embodiments, the red channel of the final output lock state 620-2 includes a non-negative value for the remaining lifetime. Thus, when the final output lock state 620-2 becomes the input lock state 620-1 of the next frame, the remaining lifetime stored in the red channel is also a non-negative value.

[0079] The green channel of the lock state 620 contains the current luminance of the current frame 502 at the time the pixel is locked (e.g., the average luma of the scene in the area around the pixel), which is obtained by the lock state 620 from the luminance texture 606. In at least some embodiments, the locking component 520 is populated into the green channel of the lock state 620 during the reprojection stage of the reprojection accumulation stage 611. The luminance information stored in the green channel of the lock state 620 currently being generated is used by the reprojection accumulation stage 611 when processing the next frame. The luminance values ​​stored in the green channel of the lock state 620 are used by the reprojection accumulation stage 611 as part of shading change detection for the current frame 502, so that if there is a discontinuous change in the appearance of the pixel (e.g., a sudden change in the shading of the pixel), the upscaler 332 can unlock the pixel. In at least some embodiments, the blue channel of the lock state 620 contains a confidence / reliability coefficient indicating how reliable the associated lock is. Please understand that the remaining lock lifetime information, current brightness information, and confidence / reliability coefficient can each be interchangeably stored in any of the red, green, or blue channels of the lock state 620.

[0080] In one or more embodiments, the locking component 520 determines whether a lock should be generated for a pixel in the current frame 502 by comparing a neighborhood (e.g., 3x3 pixels) of luminance values ​​associated with the pixel against a luminance difference threshold. In other words, the locking component 520 determines whether the relative luminance difference between the pixel and its neighborhood is less than a luminance difference threshold. The luminance difference can be characterized by max(A,B) / min(A,B), where A and B are luminance / lumen values. One embodiment of the luminance difference threshold is a predetermined percentage of similarity to the central luminance of the pixel neighborhood (e.g., 3x3). The locking component 520 retrieves the luminance values ​​from the adjusted color buffer 608. By using neighborhoods, the locking component 520 can detect fine features (e.g., wires or chain-linked fences) in the current frame 502 that should be locked to preserve the details of the final super-resolution frame / image. If the shading change difference (e.g., luminance difference) between the pixel and its neighboring luminance values ​​satisfies the luminance change threshold, the locking component 520 locks the pixel by generating an entry in the output lock state 620-2. For example, if the luminance change is less than (or equal to) the luminance change threshold, the locking component 520 locks the pixel. Otherwise, the locking component 520 maintains the pixel's current lock state (e.g., unlock state), which in at least some embodiments is also reflected in the output lock state 620-2. More specifically, the locking component 520 compares the luminance value of the center pixel in a 3x3 region with the luminance values ​​of the neighboring pixels in a 3x3 (or other) neighborhood. If the relative luminance difference between the center pixel and the neighboring pixels is less than the luminance difference threshold, the locking component 520 sets a bit in the mask. The locking component 520 considers the center pixel a candidate for locking if its luminance is outside the luminance range of all neighboring pixels. After the locking component 520 compares the brightness of the central pixel with all neighboring pixels, the locking component 520 determines whether the resulting mask contains noise, that is, whether it does not have a 2x2 (or other) region where all bits are set.If the mask contains noise and the luminance of the central pixel is outside the luminance range of all neighboring pixels, the locking component 520 generates a lock on the central pixel and stores the lock in the output lock state 620-2.

[0081] In addition to generating new locks, the locking component 520 (or another component of the upscaler 332) also updates the locks in input lock state 620-1 generated for previously upscaled frames. For example, the locking component 520 reduces the red channel value of the pixel lock in input lock state 620-1 generated for previously upscaled frames by the initial pixel lock length, which is obtained by dividing it by the total length of the jitter sequence (of the color buffer / camera). For example, if the initial length of the lock is initially set to 1, subtracting 1 / jitter sequence length from the remaining lifetime of the lock will result in a remaining lifetime of 0 after one iteration of the jitter sequence. In another embodiment, the locking component 520 (or another component of the upscaler 332) dynamically reduces / subtracts the red channel value of the pixel lock in input lock state 620-1 according to the contribution of the locked pixels to the final color of the current frame. When the lock reaches zero (or another threshold), the locking component 520 considers the lock to have expired and releases it. In one or more embodiments, the initial length of the lock is modified by a confidence / confidence coefficient that indicates how confident the locking component 520 is that the associated pixel should be locked, and this coefficient affects the length for which the pixel is locked. The locking component 520 updates one or more locks of the input lock state 620-1 by releasing the lock(s) as described below.

[0082] In the reprojection accumulation stage 611, the reprojection accumulation component 522 receives, as input, an occlusion mask 618, an expanded motion vector buffer 614, a reactivity mask 602, an output buffer 506-1 from a previous frame, a current luminance texture 606, a luminance history 610, an adjusted color buffer 608, and a lock state 620. The reprojection accumulation component 522 processes this input to generate an output buffer (texture) 506-2 for the current frame 502 at the target presentation resolution / size, and further generates a reprojected pixel lock (texture) 622 from a previous frame that can be mapped to the current frame 502. As described below, the reprojection accumulation component 522 accumulates the reprojected color data from the previous frame together with the upsampled color data from the current frame 502, and stores the accumulated color data in the output buffer 506-2. In one or more embodiments, output buffer 506-2 is a texture used internally by the upscaler 332 and is different from the presentation buffer 626 generated by the sharpening stage 613. Furthermore, in one or more embodiments, output buffer 506-2 is part of output buffer 506, which is represented as an array of two additional textures (e.g., output buffer 506-1 and output buffer 506-2) consumed by the reprojection accumulation stage 611. In odd frames, output buffer 506-1 is read and output buffer 506-2 is written (or vice versa). In even frames, output buffer 506-2 is read and output buffer 506-1 is written (or vice versa).

[0083] As shown in Figure 9, the reprojection accumulating stage 611 consists of several substages. In one or more embodiments, the substages include a shading change detection substage 901, an upsampling substage 903, a reprojection substage 905, a lock update substage 907, a color correction substage 909, one or more tone mapping substages 911 (illustrated as tone mapping substages 911-1 and 911-2), an accumulation substage 913, and an inverse tone mapping substage 915. In at least some embodiments, the reprojection accumulating component 522 includes one or more subcomponents 902-914 that perform operations associated with one or more of the reprojection accumulating substages 901-913. In one or more embodiments, the reprojection accumulating stage 611 is executed on a frame-by-frame basis, and the data flow illustrated in Figure 9 is executed in parallel for each pixel, once per output / presentation resolution pixel, and further executed in parallel for each pixel of the upscaled frame 508. In Figure 9, the "2x2 bilinear" reference indicates that bilinear sampling is performed at the corresponding stage, the "5x5 Lanczos" reference indicates that Lanczos resampling is performed at the corresponding stage, and the "1" reference indicates that point sampling is performed. However, it should be understood that other types of sampling are similarly applicable.

[0084] As described below, the reprojection cumulative component 522 performs multiple operations through one or more of the subcomponents 902-914, and these operations are as follows: For the current frame 502, the color buffer 320-1 is upsampled, and the (history) color data 922 and the pixel lock provided by the output buffer 506-1 of the previous frame are reprojected as they would appear from the current camera's viewpoint. The process involves cleaning the reprojected color data 922, and accumulating the final historical color data / values ​​928 (Figure 9) and the upsampled color data 918 for the current frame 502. This includes, as appropriate, inverse tone mapping the super-resolution color values ​​930 generated by the reprojection accumulation stage 611 to sharpen them.

[0085] At the start of the reprojection accumulation stage 611, the shading change detection substage 901 receives the luminance history 610, the current luminance texture 606, and the input lock state 620-1. The shading change detection component 902 processes these inputs to evaluate each pixel in the current frame 502 and detect changes in the pixel's shading. In one or more embodiments, the shading change detection component 902 also performs texture filtering, such as bilinear filtering, on these inputs. In one embodiment, the shading change detection component 902 uses the lock state 620-1 of the current pixel in a previously upscaled frame to determine whether the pixel is locked or unlocked. If the pixel is locked, the shading change detection component 902 determines whether the pixel's shading has changed by comparing the pixel's luminance (Y) value at the time the lock was generated with a shading change threshold. In one or more embodiments, the shading change detection component 902 compares the pixel's luminance with the average luminance of its neighborhood (e.g., 32x32) in the current frame 502. If the difference between the brightness of a pixel and the average brightness of neighboring pixels satisfies the shading change threshold (for example, if it is greater than or equal to it), the shading change detection component 902 determines that the pixel's shading has changed. In one or more embodiments, the shading change detection component 902 obtains the brightness of a pixel from the green channel of the pixel's input lock state 620-1. If the pixel is unlocked, in one or more embodiments, the shading change detection component 902 determines whether the pixel's shading has changed based on the brightness value of the pixel in the current frame and the historical brightness values ​​(or multiple values) of the neighbors of the pixel or past pixels. For example, if the difference between the brightness value of the pixel in the current frame and the historical brightness values ​​(or multiple values) of the pixel (or neighbors of a historical pixel) satisfies the shading change threshold, the shading change detection component 902 determines that the pixel's shading has changed.In one or more embodiments, the shading change detection component 902 obtains the current luminance value of a pixel from the current luminance texture 606 and the historical luminance value of a pixel from the luminance history 610. The shading change detection component 902 generates shading change data 924, which includes, for example, bits / flags indicating whether the shading of a pixel has changed. The shading change data 924 is received as input by the upsampling substage 903, the lock update substage 907, and the color correction substage 909.

[0086] The upsampling component 904 of the upsampling substage 903 receives shading change data 924 from the shading change detection component 902 and the adjusted color buffer 608 as input and upsamples the adjusted color buffer 608. In one or more embodiments, the upsampling component 904 uses the shading change data to modify the shape of the filter kernel, which may result in lower or higher sample weights used in the cumulative substage 913. Upsampling the adjusted color buffer 608 involves interpolating between existing pixels in the adjusted color buffer 608 to obtain estimates of their values ​​at new pixel locations. For example, if the current frame 502 has been upsampled from 1920×1080 pixels to 3840×2160 pixels, the upsampling component 904 interpolates between the original 1920×1080 pixels to estimate the color values ​​of the new pixels upsampled to the higher 3840×2160 pixel resolution. In at least some embodiments, the upsampling component 904 implements Lanczos resampling to upscale the pixels of the adjusted color buffer 608, but one or more different scaling methods or algorithms, or combinations thereof, such as sinc resampling, nearest neighbor interpolation, bilinear algorithms, bicubic algorithms, and box sampling, can also be applied.

[0087] Generally, Lanczos resampling is an interpolation method used to calculate new values ​​for digitally sampled data. When used in resizing digital images, the Lanczos function indicates which pixels in the rendered (original) image make up each pixel in which portion of the upsampled image. For example, Figure 10 shows several pixels 1002 rendered from the current frame 502. In each rendered pixel 1002, point P represents a set of low-resolution samples 1004 that are resampled and for which Lanczos weights are calculated. Each point S within pixel 1002 represents a target resolution pixel 1006 (also referred to herein as the upsampled pixel 1006) to be calculated based on resampling from the set of low-resolution samples 1004. The location of the upsampled pixel 1002-1 serves as the center of a Lanczos resampling kernel 1008, such as a 5x5 Lanczos resampling kernel represented by a dashed grid in Figure 10. The Lanczos function 1010 (illustrated as Lanczos function 1010-1 and Lanczos function 1010-2) is centered on the upsampled pixel 1006-1 being calculated. Therefore, the color value of the upsampled pixel 1006-1 is determined by applying the Lanczos resampling kernel 1008 (e.g., Lanczos(x,2)) to the low-resolution samples 1004-1 in the 5x5 neighborhood (grid) surrounding the upsampled pixel 1006-1. In other words, the weight of each low-resolution sample 1004 in the 5x5 pixel neighborhood is determined using Lanczos resampling and the distance from the low-resolution sample 1004 to the target resolution pixel 1006-1. The final color value of the upsampled pixel 1006-1 is based on the sum of the weights of each low-resolution sample 1004 in the 5x5 pixel neighborhood. In some cases, Lanczos resampling may introduce ringing artifacts.Therefore, in one or more embodiments, the final color values ​​of the upsampled pixels 1006-1 are clamped, for example, using a central 2x2 kernel range to reduce ringing artifacts.

[0088] In the above embodiment, a 5x5 pixel neighborhood is implemented during the Lanczos resampling process; however, in some cases, a 4x4 neighborhood is sampled because the weighted contribution of pixels on the outer perimeter of the 5x5 neighborhood is zero. Furthermore, in one or more embodiments, the implementation of the Lanczos kernel varies depending on the GPU being implemented. For example, in one or more GPU embodiments, a lookup table (LUT) can be used to encode the sinc(x) function of the Lanczos kernel. Using a LUT allows for a balance between arithmetic logic (ALU) operations and memory usage in the reprojection accumulation stage 611. For example, for a given jitter, the Lanczos values ​​of the samples used are the same for all pixels in the frame. Therefore, in one or more embodiments, these Lanczos values ​​are pre-calculated and recorded in the LUT. The LUT is passed to a shader that implements the upsampling component 904 as a texture. Instead of repeatedly calculating the Lanczos values, the pre-calculated Lanczos values, which may reside in a cache, are repeatedly read within the shader by the upsampling component 904. In one or more embodiments, to reduce ALU usage in the shader core, the LUT is read through a fixed-function sampling block. Implementing the LUT is faster than using ALU cycles when the shader stage bandwidth is not limited.

[0089] In at least some embodiments, the upsampling component 904 also calculates a YCoCg bounding box 1102 (Figure 11) for each pixel being upsampled. For example, Figure 11 shows a YCo bounding box 1102 constructed from a 3x3 neighborhood of pixels 1104 centered on the current pixel (e.g., pixel 1002-1 in Figure 10). It should be understood that the bounding box 1102 also has a third dimension for Cg (not shown for illustrative purposes). The bounding box 1102 is used in the color correction substage 909 described below. In one or more embodiments, the upsampling component 904 generates a bounding box 1102 for each pixel in the current frame 502 by calculating the minimum and maximum values ​​for each channel of all pixels in the neighborhood (e.g., 3x3) surrounding the current pixel. Different shading / patterns of the square 1106 within the 3x3 pixel neighborhood 1104 represent the Y value of the pixel. The position of each square within the bounding box 1102 represents where the sample is located within the bounding box 1102. In at least some embodiments, the output of the upsampling substage 903 includes upsampled (pixel) color data / values ​​918 for the upsampled pixels generated for the current frame 502 (i.e., pixels upsampled to the presentation resolution), and includes a YCoCg bounding box 1102 generated for each upsampled pixel.

[0090] The reprojection component 906 of the reprojection substage 905 receives the expanded motion vector buffer 614, the input lock state 620-1, and the output buffer 506-1 of the previously upscaled frame (i.e., the frame previously processed by the upscaler 332) as input. The reprojection component 906 processes this input for pixels that do not have new locks generated for the current frame 502, and reprojects the output color buffer from the output buffer 506-1 and the pixel lock information from the input lock state 620-1 of the previous frame as it would appear from the camera's viewpoint for the current frame 502. For example, Figure 12 shows frame 1202, which represents a previously upscaled frame. The reprojection component 906 samples the expanded motion vectors in the expanded motion vector buffer 614. Next, the reprojection component 906 applies the sampled motion vectors 1204 (illustrated as motion vectors 1204-1 and 1204-2) to the output buffer 506-1 (e.g., color data) of the previously upscaled frame. Figure 12 shows the sampled two-dimensional motion vectors 1204 applied to the position of the current pixel 1206 from the output buffer 506-1 of the previously upscaled frame. That is, the reprojection component 906 obtains the historical coordinates by subtracting the sampled motion vectors 1204 from the position coordinates of the current pixel. Next, the reprojection component 906 uses the historical coordinates (the transformed pixel position) to sample the input lock state 620-1 and historical color information (from the output buffer 506-1).

[0091] In one or more embodiments, the reprojection component 906 performs Lanczos resampling to sample the neighborhood of pixels surrounding the transformed pixel location. For example, Figure 12 shows a Lanczos(x,2) resampling kernel 1208 applied to a 5x5 grid 1210 of pixels surrounding the transformed pixel location 1206-1. In one or more embodiments, the reprojection component 906 reprojects the lock using bilinear interpolation at the transformed pixel location. Thus, the output of the reprojection substage 905 is a presentation resolution image containing all the data from the previous frame that can be mapped to the current frame 502. In other words, the output of the reprojection substage 905 is the reprojected color data 922 and the reprojected pixel locks 622 of pixels in the previously upscaled frame that can be mapped to the current frame 502. For example, in the input lock state 620-1 of a pixel for which no new lock has been generated for the current frame 502, the remaining lifetime from the red channel, the current luminance of the current frame 502 from the green channel, and the confidence / reliability coefficient from the blue channel (if implemented) are reprojected / written as a reprojected lock 622. In one or more embodiments, the reprojected lock 622 refers to the reprojected lock information written to the pixel's output lock state 620-2, or the reprojected lock information stored in an intermediate lock texture. In at least some embodiments, the reprojection component 906 determines whether a new lock has been generated for a pixel based on the pixel's output lock state 620-2, which includes a negative remaining lifetime value. In one or more embodiments, if the input lock state 620-1 of a pixel indicates that the pixel has been locked only once (for example, if the remaining lifetime value is "1"), the lock is not active, and the reprojection component 906 does not perform a reprojection operation on the pixel.In at least some embodiments, the reprojected pixel lock 622 is reprojected to output lock state 620-2 and used as a lock for pixels in the current frame 502 that have not been updated with new sample information, while the pixel lock generated during the locking stage 609 is used for pixels that have updated samples within that region. It should be understood that the lock reprojection process of the reprojection substage 905 is not limited to image upscaling. For example, the lock reprojection process can be applied to TAA and other techniques.

[0092] The lock update component 908 of the lock update substage 907 receives the de-occlusion mask 618, the reprojected lock 622, and the shading change data 924 as input. The lock update component 908 processes this input to generate an updated set of pixel locks 926. Updating the pixel locks here involves identifying from the reprojected lock 622 the set of pixel locks that should be passed to the color correction substage 909. For example, if the lock update component 908 determines that the de-occlusion mask 618 indicates that a pixel associated with a reprojected pixel lock has been occluded, the lock update component 908 unlocks it. In another embodiment, if the lock update component 908 determines that the shading of a pixel associated with a reprojected pixel lock has changed beyond a shading change threshold, the lock update component 908 unlocks it. In one or more embodiments, the lock update component 908 updates the reprojected locks to reflect the attenuated and unlocked locks.

[0093] Next, the lock update component 908 determines which of the remaining reprojected locks are reliable for the current frame 502. In one or more embodiments, the lock update component 908 determines the reliability of a pixel lock by comparing the luminance value associated with the pixel lock (i.e., the luminance value of the pixel at the time the pixel lock was generated) with the luminance value of the current luminance texture 606, which represents the average luminance of the scene in the vicinity of the pixel centered on the pixel associated with the current pixel lock. If the luminance separation between the compared luminance values ​​exceeds (or is equal to) the luminance separation threshold, the lock update component 908 determines that the pixel lock is unreliable and unlocks it. However, if the luminance separation between the compared luminance values ​​falls below (or is equal to) the luminance separation threshold, the lock update component 908 determines that the pixel lock is reliable and maintains the pixel lock.

[0094] In one or more embodiments, the lock update component 908 processes the input to identify new pixel locks generated during the locking stage 609 of the current frame 502 that are not reprojected. As described above, in at least some embodiments, the lock update component 908 identifies new pixel locks based on whether the red channel of the output lock state 620-2 has a negative value or whether the remaining lock lifetime is equal to the initial lock lifetime. If a new pixel lock is identified, the lock update component 908 does not perform the unlock operation described above for this new pixel lock. However, in one or more embodiments, the lock update component 908 updates the green channel of the output lock state 620-2 with current luminance information from the current luminance texture 606 for new active pixel locks (e.g., pixel locks corresponding to at least the second pixel lock for a pixel). In one or more embodiments, the lock update component 908 updates the confidence / confidence coefficients of the new active pixel locks and the pixel locks stored in the blue channel (if implemented) of the lock state 620 of the reprojected pixel locks. For example, in at least some embodiments, the lock update component 908 increments the confidence / confidence coefficients for the new pixel lock and the reprojected lock based on the current upsampling weight contribution of the associated pixel. The increment is within the normalization range [0,1] and is determined by dividing the upsampling weight of the current frame (e.g., Lanczos weight calculated in the upsampling substage 903) by a constant value, e.g., the maximum cumulative weight. This increment is added to the current confidence / confidence coefficient stored in the blue channel of the lock state 620. In at least some embodiments, the confidence / confidence coefficient is clamped to the range [0,1].

[0095] In one or more embodiments, the lock update component 908 reduces the lock lifetime of new pixel locks and reprojected locks. For example, the lifetime of a reprojected lock, or new pixel lock, is reduced by approximately 1 / 32 (in the example of 2x upscaling). In another embodiment, the lifetime of a reprojected lock, or new pixel lock, is dynamically reduced according to the contribution of the locked pixel to the final color of the current frame, i.e., the current upsample weight contribution. In at least some embodiments, the block update component 908 calculates the reduction by dividing the upsample weight of the current frame by a coefficient based on the jitter sequence length and the average upsample kernel weight. In one or more embodiments, this reduction is within the normalization range [0,1]. The lock update component 908 then updates the remaining lifetime of the pixel lock stored in the red channel of the lock state 620 by subtracting the reduction from the current remaining lifetime of the pixel lock. In one or more embodiments, the resulting value is clamped to the range [0,1] to avoid negative lifetime values.

[0096] The output of the lock update substage 907 is either a set of updated locks 926 (i.e., lock states 620) including reprojected pixel locks 622 determined to be reliable, or a set of reprojected locks 622 having an indicator associated with each lock that identifies whether the lock is reliable or unreliable. In one or more embodiments, the set of updated pixel locks 926 includes newly generated and active pixel locks for the current frame 502. In at least some embodiments, each reprojected pixel lock in the set of updated locks 926 includes updated remaining lifetime, luminance value, and confidence / reliability coefficient, and further, each newly generated active lock includes updated luminance value and confidence / reliability coefficient as described above. The updated output lock states 620-2 for the new pixel locks are then stored in the output buffer 506-2 for use during upsampling of the next frame. It should be understood that the lock update substage 907 is not limited to image upscaling. For example, the lock update substage 907 can be applied to TAA and other techniques.

[0097] Next, the updated set of locks 926 is passed as input to the color correction substage 909. The color corrector component 910 of the color correction substage 909 receives as input, for example, upsampled color data 918 for the upsampled pixels of the current frame 502, color data 922 for the reprojected pixels of the previously upscaled frame, the updated set of locks 926, an occlusion mask 618, a Luma stability coefficient (e.g., the w channel of the Luma history 610), shading change data 924, and a lock contribution coefficient for the newly locked or reprojected locked pixels. The lock contribution coefficient is calculated as a function of the weights of the upsampled color data 918, an arbitrary transparency and composition mask, and an arbitrary reactivity mask 602. The color corrector component 910 processes the input to determine the final history color data / value 928 for each upsampled pixel of the current frame 502 and its contribution to the final upscaled pixel. In one or more embodiments, when determining the final historical color value 928 for each upsampled pixel, the color corrector component 910 reduces the influence of historical samples based on various factors, including, among others, locked state or deocclusion. In other words, the color corrector component 910 modifies the historical color value in some cases to match the shading of the current frame 502. For example, if a pixel position is (partially) deoccluded, it means that the pixel position was occluded in a previously upscaled frame. Therefore, the historical color data for the pixel position is no longer effective in improving the quality of the current pixel. Hence, the color corrector component 910 reduces the influence of historical samples of deoccluded pixels to avoid artifacts such as ghosting. In this embodiment, the color corrector component 910 reduces the influence of historical samples by modulating the color value of the historical samples with a deocclusion mask 618.

[0098] Figure 13 shows another example in which the correction component 910 reduces the influence of the hierarchical sample (Sh) 1302 based on the distance (d) of the upsampled pixel from the color bounding box 1102. In this embodiment, the greater the distance (d), the greater the influence of the hierarchical sample 1302 is reduced by the color corrector component 910. In one or more embodiments, if the color corrector component 910 determines that the hierarchical sample is locked based on the updated set of locks 926, and further determines that the reactive mask 602 or the de-occlusion mask 618 does not invalidate the lock, the color corrector component 910 increases the influence of the hierarchical color data of the hierarchical sample on the upsampled color data 918.

[0099] In one or more embodiments, at least one of the upsampled color data 918 or the final historical color data 928 is passed to a tone mapping substage 911 (illustrated as tone mapping substage 911-1 and tone mapping substage 911-2). For example, if the upsampled color data 918 or the final historical color data 928 includes high dynamic range data, the upsampled color data 918 or the final historical color data 928 is passed to the tone mapping substage 911, respectively. Next, a tone mapping component 912 (illustrated as tone mapping component 912-1 and tone mapping component 912-2) applies one or more local tone mapping operations to the upsampled color data 918 or the final historical color data 928 to reduce artifacts such as firefly artifacts. In one or more embodiments, tone mapping ensures that the color values ​​are in the range of 0 to 1 by dividing the color by the maximum channel value, for example, Color.rgb = color.rgb / (1 + max(max(color.r, color.g), color.b)), where max is the maximum channel value, color.r is the red channel value, color.g is the green channel value, and color.b is the blue channel value.

[0100] The cumulative component 914 of the cumulative substage 913 receives as input the upsampled color data 918 for each pixel of the current frame 502, and the final historical color data 928, which may or may not be tone-mapped. For each upsampled pixel of the current frame 502, the cumulative component 914 accumulates (e.g., blends) the upsampled color data 918 with the final historical color data 928 of the upsampled pixel to collectively form the final cumulative super-resolution color value 930 of the upsampled version of the current frame 502. The implicit consequence of the temporal accumulation of multiple frames in the cumulative substage 913 is that the resulting final cumulative super-resolution color value 930 is anti-aliased. In one or more embodiments, the cumulative component 914 blends the upsampled color data 918 from the current frame 502 with the final historical color data 928 using a relatively low linear interpolation coefficient, such that the super-resolution color value 930 of the upscaled frame 508 includes a small amount of color data from the current frame 502. In other words, the final historical color data 928 determined for each upsampled pixel contributes more to the final cumulative super-resolution color value 930 than the upsampled color data 918 from the current frame 502. The blend of the two color values ​​can be characterized as follows: Blended Value = A*(1-W)+B*W, where A is the final historical color data 928, B is the color data from the current frame 502, and W is the contribution weight. In one or more embodiments, the contribution weight of the upsampled color data 918 can be increased based on the reactive mask 602, and the cumulative component 914 also accepts this as input.The reactive mask 602 should reduce reliance on historical information when compositing the currently upsampled pixels, and instead instructs the cumulative component 914 where the color data 918 of the upsampled pixels should contribute more to the final cumulative super-resolution color value 930 (hereinafter also referred to as the “super-resolution color value 930” for brevity). For example, for a given pixel in the current frame 502, the cumulative component 914 multiplies the reactive value from the reactive mask 602 associated with the pixel with a contribution weight W in order to increase the contribution weight of the upsampled color data 918. Thus, a blend of two color values ​​considering the reactive mask 602 can be characterized as follows: Blended Value = A * (1 - (R * W) + B * (R * W). Here, A is the final historical color data 928, B is the color data from the current frame, and R is the reactivity value from the reactivity mask 602.

[0101] The super-resolution color value 930 represents an upscaled version of the current frame 502 at the presented resolution and is stored in the output buffer 506-2. In other words, the output buffer 506-2 contains the upscaled frame 508. In one or more embodiments, the output buffer 506-2 is used as the output buffer 506-1 that the upscaler 332 takes as input when processing the next frame. In one or more embodiments, the super-resolution color value 930 is passed to the inverse tone mapping substage 915 and then stored in the output buffer 506-2. In these embodiments, the inverse tone mapping component 916 inverse tone-maps the super-resolution color value 930. The inverse tone-mapping process inverts the color tone-mapping process described above so that the color range of the super-resolution color value 930 matches the color range of the input frame 502. In one or more embodiments, inverse tone mapping is characterized by Color.rgb = color.rgb / (1.f - max(max(color.r,color.g),color.b)), where f is a floating-point number, max is the maximum channel value, color.r is the red channel value, color.g is the green channel value, and color.b is the blue channel value.

[0102] Referring to Figure 6, the sharpening component 524 of the sharpening stage 613 receives an output buffer 506-2 containing the upscaled frame 508 as input. The sharpening component 524 performs one or more sharpening operations on the pixels of the upscaled frame 508 represented by the output buffer 506-2. The sharpening component 524 stores the sharpened upscaled frame 508 in the presentation buffer 626. In one or more embodiments, the sharpening component 524 performs Robust Contrast Adaptive Sharpening (RCAS). However, other image sharpening techniques are equally applicable. RCAS generates further clarity and sharpness in the final upscaled image and, when converting local contrast to a variable amount of sharpness, resolves to maximize local sharpness as much as possible before clipping. RCAS also incorporates a process to limit the sharpening of potential noise in the image. More specifically, as shown in Figure 14, RCAS operates on data 1402 (Figure 14) sampled from output buffer 506-2 using a 5-tap filter 1404 (Figure 14) composed of a cross pattern. During RCAS processing, the sharpening component 524 calculates weights w based on the following equation:

[0103]

number

[0104] The sharpening component 524 selects either w0 or w1 based on a weight w that does not cause clipping, and further restricts the weight w. The sharpening component 524 multiplies by a "sharpness" amount, which is a sharpness value provided by the application 206 that modifies the perceptual sharpness of the output image. The RCAS process performs high-pass filtering, normalizes for local contrast, and then shapes the image. This process is used as a noise detection filter to reduce the influence of RCAS on grain and focus on actual edges. Furthermore, the RCAS process also supports pass-through alpha. Once the sharpening process is complete, the sharpening component 524 stores the data of the upscaled frame 508 in the presentation buffer 626. In at least some embodiments, the post-upscaling post-processing stage 311-2 performs one or more post-processing operations on the upscaled frame 508 stored in the presentation buffer 626. The post-processed upscaled frame 508 is then output by the graphics processing pipeline 214 for presentation to the display device 118. Alternatively, the upscaled frame 508 is sharpened for presentation on the display device 118 without performing any post-upscaling operations, and then output by the graphics processing pipeline 214.

[0105] Figure 15 shows a flowchart outlining one exemplary method 1500 that performs spatial upscaling of a rendered frame 502 of a video stream by using temporal feedback to reconstruct a high-resolution image representing the rendered frame 502. The process described below with respect to method 1600 should be understood in more detail by referring to Figures 6 through 14. In block 1502, the upscaler 332 obtains the first frame of the video stream rendered at a first resolution. The first frame is defined by a first plurality of pixels associated with a first set of color data. In block 1504, the upscaler obtains the second frame of the video stream upscaled to a second resolution higher than the first resolution. The second frame is defined by a second plurality of pixels associated with a second set of color data. In block 1506, the upscaler locks one or more pixels of the first plurality of pixels. In block 1508, the upscaler 332 upsamples the first plurality of pixels to the second resolution. This upsampling generates upsampled color data for the first set of upsampled pixels based on the first set of color data. In block 1510, the upscaler 332 accumulates the upsampled color data with the second set of color data to generate the final color data for the first set of upsampled pixels. The upsampled color data associated with one or more locked pixels is maintained during the accumulation of upsampled color data for at least one of the rendered frame or the next frame. Furthermore, the color data of the second set of color data associated with the pixel locks contributes to the final color data more than the corresponding color data of the upsampled color data. In block 1512, the upscaler 332 stores the first set of upsampled pixels, along with the final color data, as an upscaled frame representing the first frame at the second resolution.

[0106] Figures 16 and 17 show in flowchart form an overview of another exemplary method 1600 that performs spatial upscaling of a rendered frame 502 of a video stream by using temporal feedback to reconstruct a high-resolution image representing the rendered frame 502. The process described below with respect to method 1600 should be understood in more detail by referring to Figures 6 through 14. In block 1602, the rendering stage 313 of the graphics pipeline 214 renders frame 502 at a first resolution. In block 1604, the rendered frame 502 is passed to the super-resolution upscaler 332 of the graphics pipeline 214. In block 1606, the auto-exposure stage 601 of the upscaler 332 processes the color buffer 320-1 of the rendered frame 502 to generate an exposure texture 604 containing the exposure value(s) determined for the current frame, and a current luminance texture 606 containing the current luminance value(s) for the rendered frame 502. In block 1608, the input color adjustment stage 603 of the upscaler 332 processes the exposure texture 604 and the color buffer 320-1 of the rendered frame 502 to generate the adjusted color buffer 608 and the luminance history texture 610 containing historical luminance information / values ​​for the rendered frame 502.

[0107] In block 1610, the reconfiguration and expansion stage 605 of upscaler 332 processes the depth buffer 322-1 and motion vector buffer 504 of the rendered frame 502 to produce the previous depth buffer 616, the expanded depth buffer 612, and the expanded motion vector buffer 614 of the rendered frame 502. In block 1612, the depth clipping stage 607 of upscaler 332 processes the previous depth buffer 616, the expanded depth buffer 612, and the expanded motion vector buffer 614 to produce an occlusion mask 618 for the rendered frame 502. In block 1614, the pixel locking stage 609 of upscaler 332 processes the adjusted color buffer 608 to lock one or more pixels of the rendered frame 502 and produce a locked state texture 620.

[0108] In block 1616, the reprojection accumulation stage 611 of the upscaler 332 processes the deocclusion mask 618, the expanded motion vector buffer 614, the reactive mask 602 associated with the pixels of the rendered frame 502, the output buffer 506-1 for the previously upscaled frame, the current luminance texture 606, the luminance history texture 610, the adjusted color buffer 608, and the locked state texture 620 to produce a super-resolution (and anti-aliased) upscaled frame 508 representing the rendered frame. The upscaled frame 508 has a second resolution greater than the first resolution of the rendered frame 502. In block 1618, the sharpening stage 613 of the upscaler 332 sharpens the upscaled frame 508. In block 1620, the sharpened upscaled frame 508 is stored in the presentation buffer 626 for presentation to the display device 118. In one or more embodiments, further post-processing is performed on the upscaled frame 508 before presenting it to the display device 118. The flow returns to block 1602, where the next rendered frame is processed by the upscaler 332.

[0109] Figures 18 and 19 both show a flowchart of a more detailed method 1800 of the reprojection accumulation process shown in block 1616 of Figure 17. The process described below for method 1800 should be understood in more detail by referring to Figures 9 through 14. In block 1802, the shading change detection substage 901 processes the locked state texture 620, the current luminance texture 606, and the luminance history texture 610 to generate shading change data 924 for each pixel of the rendered frame 502. In block 1804, the upsampling substage 903 processes the adjusted color buffer 608 to upsample the pixels of the rendered frame 502 to a second resolution (i.e., the target presentation resolution). The upsampling substage 903 generates a color bounding box 1102 for each upsampled pixel. In block 1806, the reprojection substage 905 processes the expanded motion vector buffer 614, the locked state texture 620, and the output buffer 506-1 of the previously upscaled frame to reproject the pixel data (e.g., color data 922) and pixel locks 622 from the previously upscaled frame, which can be mapped to the rendered frame 502.

[0110] In block 1808, the lock update stage 907 processes the occlusion mask 618, the reprojected pixel locks 622, and the shading change data 924 to generate an updated set of pixel locks 926, including trusted locks. Locks determined to be untrustworthy are released. In block 1810, the color correction substage 909 processes the upsampled pixel data (e.g., color data 918), the color bounding boxes 1102 of the upsampled pixels, the updated set of pixel locks 926, the occlusion mask 618, the color data 922 of the reprojected pixels, and the shading change data 924 to determine the final historical color data / value 928 for each upsampled pixel.

[0111] In block 1812, one or more tone mapping substages 911 tone-map the final historical color data 928 and the color data 918 of the upsampled pixels. However, in at least some embodiments, the tone mapping operation is optional. In block 1814, the cumulative substage 913 processes the final historical color data 928 and the potentially tone-mapped color data 918 to generate the final cumulative super-resolution color value 930 for the upsampled pixels. In block 1816, the inverse tone-mapping substage 915 inverse tone-maps the super-resolution color value 930. However, in at least some embodiments, the inverse tone-mapping operation is optional. In block 1818, the potentially inversely tone-mapped super-resolution color value 930 is stored in the output buffer 506-2, having a second resolution and representing an upscaled frame 508 corresponding to the rendered frame 502.

[0112] Figure 20 illustrates the overall process for generating pixel locks for one or more pixels in flowchart form. The process described below with respect to Method 2000 should be understood in more detail by referring to Figure 9. In block 2002, the locking stage 609 acquires the first frame of the video stream. The first frame is defined by a first plurality of pixels associated with a second of the color data. In block 2004, the locking stage 609 determines that the pixels of the first plurality of pixels contain high-frequency information. In block 2006, in response to the determination that the pixels contain high-frequency information, the locking stage 609 generates a first pixel lock for this pixel so that the color data associated with the pixel is maintained during the color accumulation process of the next frame. In block 2008, the locking stage 609 acquires pixel lock information for the second frame of the video stream. The second frame is defined by a second plurality of pixels. The pixel lock information identifies at least a second pixel lock associated with at least one of the pixels of the second plurality of pixels. In block 2010, locking stage 609 reduces the remaining lifespan of at least the second pixel lock.

[0113] Figure 21 shows a flowchart of a more detailed method 2100 of the pixel locking process shown in block 1614 of Figure 17. For the process described below regarding method 2100, please refer to Figure 9 for a more detailed understanding of what is described above. In block 2102, the locking stage 609 of the upscaler 332 receives the adjusted color buffer 608 and the lock state texture 621 as input for the previously upscaled frame. In block 2104, for each pixel of the rendered frame 502, the locking stage 609 compares the defined neighborhood of luminance values ​​obtained from the adjusted color buffer 608 with the luminance threshold. In block 2106, the locking stage 609 determines whether the luminance threshold is met. In block 2108, if the luminance threshold is not met (for example, if the luminance change between neighboring pixels and the current pixel exceeds the luminance threshold), the locking stage 609 does not generate a lock for the current pixel. The process proceeds to block 2106. In block 2110, if the luminance threshold is not met (for example, if the luminance change between neighboring pixels and the current pixel falls below the luminance threshold), the locking stage 609 generates a lock for the current pixel. In block 2112, as part of creating the lock, the locking stage 609 stores the remaining lifetime of the lock in the red channel of the current pixel's lock state texture 620. At the time the lock is first generated, the pixel's lifetime is fully utilized. In block 2114, the locking stage 609 stores the pixel's luminance value at the time the lock was generated in the green channel of the current pixel's lock state texture 620. In block 2116, for each locked pixel in a previously upscaled frame, the locking stage 609 updates the remaining lifetime of the lock in the red channel of that pixel's lock state texture 620.

[0114] Figure 22 shows a flowchart of Method 2200, a more detailed method for reprojection shown in block 1806 of Figure 18, and the lock update process shown in block 1808 of Figure 18. The process described below regarding Method 2200 should be understood in more detail by referring to Figures 9 and 12. In block 2202, the reprojection substage 905 samples the augmented motion vectors in the augmented motion vector buffer 614. The reprojection substage 905 applies the samples to the output buffer 506-1 of the previously upscaled frame to generate reprojected color data 922 for the pixels of the previously upscaled frame. In block 2204, the reprojection substage 905 samples the augmented motion vectors and applies the samples to the lock state texture 620 of the previously upscaled frame to generate reprojected locks 622 for the pixels of the previously upscaled frame. As described above, the expanded motion vector does not apply the lock state texture 620 of the previously upscaled frame to the lock state texture 620 of the rendered (current) frame for pixels where a new pixel lock was generated in the rendered frame. In block 2206, the reprojection substage 905 passes the reprojected color data 922 as input to the color correction substage 909. In block 2208, the reprojection substage 905 passes the reprojected lock 622 as input to the lock update substage 907.

[0115] In block 2210, for each reprojected pixel lock, the lock update substage 907 determines, based on the de-occlusion mask 618, whether the associated pixel is occluded. In block 2212, if the pixel is occluded, the lock update substage 907 releases the reprojected lock and then processes the next reprojected pixel lock. In block 2214, if the pixel is not occluded, the lock update substage 907 maintains the reprojected pixel lock. Alternatively, or in addition to block 2212, the lock update substage 907 determines in block 2216 whether the pixel shading has changed beyond the shading change threshold. In block 2218, if the pixel shading has changed beyond the shading change threshold, the lock update substage 907 releases the reprojected lock and then processes the next reprojected pixel lock. In block 2220, if the pixel shading has not changed beyond the shading change threshold, the lock update substage 907 maintains the reprojected pixel lock. In block 2222, the lock update substage 907 determines that the lock is reliable. In block 2224, the lock update substage 907 updates any active pixel locks generated for the rendered frame, for example, by updating the luminance stored in the green channel and updating the confidence / confidence coefficient stored in the blue channel. Once all reprojected and active pixel locks for the rendered frame have been processed, the lock update substage 907 passes the set of updated pixel locks 926 to the color correction substage 909.

[0116] In some embodiments, the above-described apparatus and techniques are implemented in a system comprising one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips). In at least some embodiments, electronic design automation (EDA) and computer-aided design (CAD) software tools can be used to design and manufacture these IC devices. These design tools are typically represented as one or more software programs. One or more software programs include computer-executable code for operating a computer system to operate with code representing the circuit of one or more IC devices in order to perform at least part of the process of designing or adapting a manufacturing system for manufacturing the circuit. This code may include instructions, data, or combinations of instructions and data. Software instructions representing the design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of designing or manufacturing an IC device is stored in and accessed from the same or different computer-readable storage medium.

[0117] Computer-readable storage media include any non-temporary storage media or combination of non-temporary storage media that are accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray® discs), magnetic media (e.g., floppy disks, magnetic tapes, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS) based storage media. Computer-readable storage media (e.g., system RAM or ROM) may be built into the computing system, computer-readable storage media (e.g., magnetic hard drives) may be permanently mounted to the computing system, computer-readable storage media (e.g., optical disks or Universal Serial Bus (USB) based flash memory) may be detachably mounted to the computing system, and computer-readable storage media (e.g., network-accessible storage (NAS)) may be connected to the computer system via a wired or wireless network.

[0118] In some embodiments, certain aspects of the technology described above are implemented by one or more processors of a processing system that executes the software. The software includes one or more sets of executable instructions, which are stored in a non-temporary computer-readable storage medium or otherwise clearly embodied. The software may also include instructions and specific data, which, when executed by one or more processors, operate the one or more processors to execute one or more aspects of the technology described above. Non-temporary computer-readable storage mediums may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, caches, random-access memory (RAM), or other non-volatile memory devices (one or more). Executable instructions stored in a non-temporary computer-readable storage medium can be implemented as source code, assembly language code, object code, or other instruction forms that can be interpreted or otherwise executed by one or more processors.

[0119] In addition to the foregoing, it should be noted that not all activities or elements described in the summary are required, and certain activities or parts of devices may not be required, and one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed does not necessarily indicate the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and variations can be made without departing from the scope of the invention as described in the claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive, and all of these variations are intended to fall within the scope of the invention.

[0120] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and features that may give rise to or manifest any benefits, advantages, or solutions are not to be construed as essential, necessary, or indispensable features to any or all of the claims. Furthermore, the disclosed invention can be modified and implemented in different but similar ways, in ways that are obvious to those skilled in the art who are interested in the teachings of this specification; therefore, the specific embodiments described above are merely illustrative. There are no limitations to the details of the configuration or design shown herein beyond those described in the appended claims. Accordingly, the specific embodiments described above may be modified or altered, and it is clear that all such modifications are within the scope of the disclosed invention. Accordingly, the protection sought herein is described in the appended claims.

Claims

1. A method in an acceleration processing device, Obtaining a first frame of a video stream rendered at a first resolution, wherein the first frame is defined by a first plurality of pixels associated with a first set of color data, Obtaining a second frame of the video stream that has been upscaled to a second resolution higher than the first resolution, wherein the second frame is defined by a second plurality of pixels associated with a second set of color data, The first plurality of pixels are upsampled to the second resolution, wherein the upsampling is performed by generating upsampled color data for the first plurality of pixels to be upsampled based on the first set of color data. The upsampled color data is accumulated with a second set of color data to generate final color data for a first plurality of upsampled pixels, wherein the color data of the second set of color data associated with the pixel lock contributes more to the final color data than the corresponding color data of the upsampled color data. This includes storing the upsampled first plurality of pixels together with the final color data as an upscaled frame representing the first frame at the second resolution, method.

2. Based on the first set of color data, the exposure value of the first frame is determined, The further includes determining the current brightness of the first frame based on the first set of color data, The method according to claim 1.

3. Determining the exposure value includes downsampling the first set of color data to store the exposure value at a 1x1 mipmap level, and determining the current brightness includes downsampling the first set of color data to store the current brightness at a mipmap level with half the resolution. The method according to claim 2.

4. The method further includes locking one or more of the first plurality of pixels and converting the first set of color data from a first color space to a second color space before upsampling the first plurality of pixels. The method according to claim 1.

5. The depth values ​​in the depth buffer of the first frame are expanded to generate the expanded depth buffer, The motion vectors in the motion vector buffer of the first frame are expanded to generate the expanded motion vector buffer, Based on the expanded depth buffer and the expanded motion vector buffer, the depth value of each pixel in the second frame is estimated for each of the first plurality of pixels. This further includes storing the estimated depth value in a previous depth buffer, The method according to claim 1.

6. The method further includes generating a de-occlusion mask for each of the first plurality of pixels based on the expanded depth buffer, the expanded motion vector buffer, and the previous depth buffer, wherein the de-occlusion mask indicates the amount by which the pixel was de-occluded in the second frame. The method of claim 5.

7. For each of the first plurality of pixels, the amount of shading change of the pixel between the first frame and the second frame is determined. For each of the first plurality of pixels, the method further includes generating shading change data based on the amount of shading change determined for that pixel. The method according to claim 1.

8. Determining the amount of shading change for the aforementioned pixel is: Depending on whether the pixel is locked, the brightness of the pixel at the time the lock was generated is compared with a shading change threshold, Depending on whether the pixel is unlocked, the current brightness of the pixel is compared with the historical brightness information associated with the second frame, The method according to claim 7.

9. Upsampling the first plurality of pixels to the second resolution includes applying a Lanczos resampling kernel to the vicinity of each of the first plurality of pixels, centered on the pixel. The method according to claim 1.

10. For each of the first plurality of pixels, the method further includes generating a color space bounding box based on the neighborhood of the pixel centered on that pixel. The method of claim 9.

11. The further step includes reprojecting the second set of color data onto the first frame to generate the reprojected color data. The method according to claim 1.

12. Reprojecting the second set of the aforementioned color data is Sampling motion vectors from the motion vector buffer associated with the first frame, This includes applying the sampled motion vector to a second set of color data from the second frame, The method according to claim 11.

13. The further step includes reprojecting the set of pixel locks associated with the second plurality of pixels to generate reprojected pixel locks, The method according to claim 11.

14. The further includes determining the final historical color value for each of the upsampled first plurality of pixels based on at least the reprojected color data of the second frame, The method according to claim 11.

15. Determining the final historical color value means that By applying the occlusion removal mask to the reprojected color data, one or more pixels that have been deoccluded in the second plurality of pixels are determined, To reduce the influence of the color data of the reprojected color data associated with one or more pixels that are determined to be free from occlusion on the final historical color value, This includes reducing the influence of the reprojected color data based on the distance of the color data from the color space bounding box generated for the first plurality of upsampled pixels, The method according to claim 14.

16. Accumulating the upsampled color data with the second set of color data is: For each of the first plurality of upsampled pixels, the upsampled color data of the pixel is accumulated with the final historical color value. The method according to claim 14.

17. It is a system, Central processing unit and An acceleration processing device comprising a graphics processing pipeline, wherein the graphics processing pipeline comprises a plurality of stages including an upscaling stage, The aforementioned upscaling stage is Obtaining a first frame of a video stream rendered at a first resolution, wherein the first frame is defined by a first plurality of pixels associated with a first set of color data, Obtaining a second frame of the video stream that has been upscaled to a second resolution higher than the first resolution, wherein the second frame is defined by a second plurality of pixels associated with a second set of color data, The first plurality of pixels are upsampled to the second resolution, wherein the upsampling is performed by generating upsampled color data for the first plurality of pixels to be upsampled based on the first set of color data. The upsampled color data is accumulated with a second set of color data to generate final color data for a first plurality of upsampled pixels, wherein the color data of the second set of color data associated with the pixel lock contributes more to the final color data than the corresponding color data of the upsampled color data. The upsampled first plurality of pixels are stored together with the final color data as an upscaled frame representing the first frame at the second resolution. It is configured to do, system.

18. The aforementioned upscaling stage is The second set of color data is reprojected onto the first frame to generate the reprojected color data, For each of the upsampled first plurality of pixels, the final historical color value is determined based on the reprojected color data of the second frame, It is configured to do, The system according to claim 17.

19. The aforementioned upscaling stage is For each of the first plurality of upsampled pixels, the upsampled color data of the pixel is accumulated with the final historical color value, thereby accumulating the upsampled color data with the second set of color data. The system according to claim 18.

20. A computer-readable storage medium that embodies a set of executable instructions for operating at least one processor, The set of instructions is Obtaining a first frame of a video stream rendered at a first resolution, wherein the first frame is defined by a first plurality of pixels associated with a first set of color data, Obtaining a second frame of the video stream that has been upscaled to a second resolution higher than the first resolution, wherein the second frame is defined by a second plurality of pixels associated with a second set of color data, The first plurality of pixels are upsampled to the second resolution, wherein the upsampling is performed by generating upsampled color data for the first plurality of pixels to be upsampled based on the first set of color data. The upsampled color data is accumulated with a second set of color data to generate final color data for a first plurality of upsampled pixels, wherein the color data of the second set of color data associated with the pixel lock contributes more to the final color data than the corresponding color data of the upsampled color data. The upsampled first plurality of pixels are stored together with the final color data as an upscaled frame representing the first frame at the second resolution. This is performed by at least one of the processors. Computer-readable storage medium.

Images