Super-resolution magnification

Abstract

A first frame of a video stream rendered at a first resolution is obtained. A second frame of the video stream that is amplified to a second higher resolution is also obtained. The first plurality of pixels is magnified to the second resolution. The up-sampling generates up-sampled color data for the up-sampled first plurality of pixels. The up-sampled color data is accumulated with a second set of color data associated with a second plurality of pixels defining the second frame to generate final color data for the up-sampled first plurality of pixels. Color data associated with a pixel lock in the second set of color data contributes more to the final color data than corresponding color data in the up-sampled color data. The up-sampled first plurality of pixels are stored together with the final color data as an amplified frame representing the first frame at the second resolution.

Description

Background Art

[0001] Spatial amplifiers implemented in a graphics pipeline typically take frames rendered at a lower than native resolution for improved performance and upscale them to the native resolution of the display. Although easy to integrate, spatial amplifiers can have certain disadvantages. For example, conventional spatial upscaling methods typically require high-quality anti-aliased source images. Therefore, games without anti-aliasing must implement anti-aliasing, which makes integrating conventional methods more time-consuming. In addition, if anti-aliasing is not implemented well, conventional spatial amplifiers will typically generate poor upscaled output. In addition, the quality of the upscaling depends on the source resolution input. When the source resolution is very low, there is not enough information to regenerate fine details, resulting in additional artifacts such as flickering and poor edge reconstruction. Summary of the invention

[0002] The present disclosure may be better understood, and its numerous features and advantages become apparent to those skilled in the art by referring to the following embodiments, alone or in combination.

[0003] Embodiment 1: A method performed at an accelerated processing device, comprising: obtaining a first frame of a video stream rendered at a first resolution, wherein the first frame is defined by a first plurality of pixels associated with a first set of color data; obtaining a second frame of the video stream that is enlarged to a second resolution greater than the first resolution, wherein the second frame is defined by a second plurality of pixels associated with a second set of color data; upsampling the first plurality of pixels to the second resolution, wherein the upsampling generates upsampled color data for the upsampled first plurality of pixels based on the first set of color data; accumulating the upsampled color data with the second set of color data to generate final color data for the upsampled first plurality of pixels, wherein color data associated with a pixel lock in the second set of color data contributes more to the final color data than corresponding color data in the upsampled color data; and storing the upsampled first plurality of pixels together with the final color data as an enlarged frame representing the first frame at the second resolution.

[0004] Embodiment 2: The method according to embodiment 1 further includes: determining an exposure value of the first frame based on the first set of color data; and determining a current brightness of the first frame based on the first set of color data.

[0005] Embodiment 3: A method according to Embodiment 2, wherein determining the exposure value comprises performing downsampling of the first set of color data to store the exposure value at a 1×1 mipmap level, and wherein determining the current brightness comprises performing downsampling of the first set of color data to store the current brightness at a half-resolution mipmap level.

[0006] Embodiment 4: The method according to any one of embodiments 1 to 3 further includes: before locking one or more pixels and upsampling the first plurality of pixels, converting the first set of color data from the first color space to the second color space.

[0007] Embodiment 5: The method according to any one of Embodiments 1 to 4 further includes: expanding the depth value in the depth buffer of the first frame to generate an expanded depth buffer; expanding the motion vector in the motion vector buffer of the first frame to generate an expanded motion vector buffer; for each pixel in the first plurality of pixels, estimating the depth value of the pixel in the second frame based on the expanded depth buffer and the expanded motion vector buffer; and storing the estimated depth value in the previous depth buffer.

[0008] Embodiment 6: The method according to embodiment 5 further includes: for each pixel in the first plurality of pixels, generating a de-occlusion mask based on the expanded depth buffer, the expanded motion vector buffer and the previous depth buffer, wherein the de-occlusion mask indicates the amount by which the pixel is de-occluded in the second frame.

[0009] Embodiment 7: The method according to any one of embodiments 1 to 6 further comprises: for each pixel in the first plurality of pixels, determining a change in shading of the pixel between the first frame and the second frame; and

[0010] For each pixel in the first plurality of pixels, color change data is generated based on the color change amount determined for the pixel.

[0011] Embodiment 8: A method according to Embodiment 7, wherein determining the shading change amount of the pixel comprises: in response to the pixel being locked, comparing the brightness of the pixel when the lock is created with a shading change threshold; and in response to the pixel being unlocked, comparing the current brightness of the pixel with historical brightness information associated with the second frame.

[0012] Embodiment 9: A method according to any one of embodiments 1 to 8, wherein upsampling the first plurality of pixels to the second resolution comprises: for each pixel in the first plurality of pixels, applying a Lanczos resampling kernel to a neighborhood of pixels around the center of the pixel.

[0013] Embodiment 10: The method according to embodiment 9 further includes: for each pixel in the first plurality of pixels, generating a color space bounding box based on the neighborhood of pixels around the center of the pixel.

[0014] Embodiment 11: The method according to any one of embodiments 1 to 10 further includes: reprojecting the second set of color data to the first frame to generate reprojected color data.

[0015] Embodiment 12: A method according to embodiment 11, wherein reprojecting the second set of color data comprises: sampling a motion vector from a motion vector buffer associated with the first frame; and applying the sampled motion vector to the second set of color data of the second frame.

[0016] Embodiment 13: The method according to Embodiment 11 further comprises: reprojecting a set of pixel locks associated with the second plurality of pixels to generate reprojected pixel locks.

[0017] Embodiment 14: The method according to embodiment 11 further includes: for each pixel in the upsampled first plurality of pixels, determining a final historical color value based at least on the re-projected color data of the second frame.

[0018] Embodiment 15: A method according to embodiment 14, wherein determining the final historical color value comprises: determining one or more pixels in the second plurality of pixels that are de-occluded by applying a de-occlusion mask to the reprojected color data; reducing the influence of color data from the reprojected color data associated with the one or more pixels determined to be de-occluded on the final historical color value; and reducing the influence of the color data based on the distance between the color data from the reprojected color data and a color space bounding box generated for the upsampled first plurality of pixels.

[0019] Embodiment 16: A method according to Embodiment 14, wherein accumulating the upsampled color data with the second set of color data comprises: for each pixel in the upsampled first plurality of pixels, accumulating the upsampled color data of the pixel with the final historical color value.

[0020] Embodiment 17: An accelerated processing device comprises a graphics processing pipeline for executing the method according to any one of embodiments 1 to 16. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By referring to the accompanying drawings, the present disclosure will also be better understood, and its numerous features and advantages will become apparent to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical items.

[0022] Figure 1 is a block diagram of an example device for implementing the super-resolution upscaling techniques described herein, according to some specific implementations.

[0023] Figure 2 is an example of additional details related to performing processing tasks on one or more accelerated processing devices according to some specific implementations Figure 1 A more detailed block diagram of an example device is shown in FIG.

[0024] Figure 3 It is implemented according to some specific Figure 2 Block diagram of the graphics pipeline.

[0025] Figure 4 A graphic pipeline integration diagram for super-resolution upscaling is illustrated according to some implementations.

[0026] Figure 5 is an example according to some specific implementations Figure 4 Block diagram of a more detailed view of the super-resolution amplifier.

[0027] Figure 6 is an example according to some specific implementations Figure 5 Block diagram of the various stages / passes of a super-resolution amplifier.

[0028] Figure 7 is an example according to some specific implementations Figure 5 Illustration of the downsampling technique for the auto-exposure component of the super-resolution amplifier.

[0029] Figure 8 is an example of a method according to some specific implementations Figure 5 Schematic diagram of an example of a dilated depth value and motion vector construction process performed by a reconstruction and dilation component of a super-resolution upscaling.

[0030] Fig. 9 is an example according to some specific implementations Figure 5 Block diagram of the various sub-stages of the reprojection and accumulation stages of the super-resolution amplifier.

[0031] Fig.10 is an example of a method according to some specific implementations Figure 5 Schematic diagram of an example resampling process performed by an upsampling component of a super-resolution amplifier.

[0032] Fig.11 is an example of a method according to some specific implementations Figure 5 An illustration of an example of a color bounding box construction process performed by an upsampling component of a super-resolution upsampling component of FIG.

[0033] Fig.12 is an example of a method according to some specific implementations Figure 5 An illustration of an example of a color data reprojection process performed by a reprojection component of a super-resolution amplifier.

[0034] Fig.13is an example of a method according to some specific implementations Figure 5 An illustration of an example of a color correction process performed by a color corrector component of a super-resolution amplifier.

[0035] Fig.14 is an example of a method according to some specific implementations Figure 5 An illustration of an example of a filter implemented by the image sharpening component of a super-resolution amplifier.

[0036] Fig.15 is a flow chart illustrating an overall method for spatially upscaling a rendered frame of a video stream by reconstructing a high-resolution image representing the rendered frame using temporal feedback, according to some implementations.

[0037] Fig.16 and Fig.17 Together are flow charts illustrating another general method for spatially upscaling a rendered frame of a video stream by reconstructing a high-resolution image representing the rendered frame using temporal feedback, according to some implementations.

[0038] Fig.18 and Fig.19 Together, the examples are based on some specific implementations of Fig.17 A flowchart of a more detailed method of the reprojection and accumulation process is illustrated in box 1616 of .

[0039] Fig. 20 is a flow chart illustrating an overall method of a pixel locking process illustrated according to some specific implementations.

[0040] Fig.21 is an example of some specific implementations of Fig.17 A flowchart of a more detailed method of the pixel locking process is illustrated in box 1614 of FIG.

[0041] Fig. 22 is an example of some specific implementations of Fig.18 The reprojection illustrated in block 1806 and in Fig.18 A flowchart of a more detailed method of the lock update process is illustrated in block 1808 of FIG. DETAILED DESCRIPTION

[0042] As described below, an example method provides improved upscaling by using temporal feedback to reconstruct a high-resolution image while maintaining and improving image quality compared to native rendering. This method can achieve "practical performance" for expensive rendering operations such as hardware ray tracing.

[0043] Figure 1 illustrative example of a method in which one or more features described herein may be implemented, such as super-resolution amplifier 332 ( Figure 3)) is an example device 100. The device 100 includes, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, a tablet computer, a wearable computing device, etc. In one or more specific implementations, the device 100 includes one or more processors 102, a memory 104, a storage device 106, an input device 108, and an output device 110. In at least some specific implementations, the device 100 also includes one or more input drivers 112 and one or more output drivers 114. It should be understood that the device 100 may include Figure 1 Additional components not shown.

[0044] In one or more specific implementations, the processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU on the same die, or one or more processor cores, where each processor core can be a CPU or a GPU. In at least some specific implementations, the CPU includes one or more single-core or multi-core CPUs. In various alternatives, the memory 104 is located on the same die as the processor 102, or is located separately from the processor 102. The memory 104 includes volatile or non-volatile memory, such as random access memory (RAM), dynamic RAM, or cache.

[0045] Storage device 106 includes fixed or removable storage devices, such as hard disk drives, solid-state drives, optical disks, or flash drives. Input device 108 includes, for example, a keyboard, a keypad, a touch screen, a touchpad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless LAN card for transmitting and / or receiving wireless IEEE 802 signals). Output device 110 includes, for example, a display, a speaker, a printer, a tactile feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless LAN card for transmitting and / or receiving wireless IEEE 802 signals).

[0046] The input driver 112 communicates with the processor 102 and the input device 108 and enables the processor 102 to receive input from the input device 108. The output driver 114 communicates with the processor 102 and the output device 110 and enables the processor 102 to send output to the output device 110. It should be noted that the input driver 112 and the output driver 114 are optional components, and if the input driver 112 and the output driver 114 are not present, the device 100 will operate in the same manner. The device 100 also includes one or more accelerated processing devices (APDs) 116. In at least some specific implementations, the APD 116 is coupled to, for example, one or more display devices 118. The APD 116 can be part of the output device 110 or separate from the output device. The APD 116 accepts computing commands and graphics rendering commands from the processor 102, processes these computing commands and graphics rendering commands, and provides pixel output to the display device 118 for display.

[0047] As described in further detail below, APD 116 includes one or more parallel processing units for performing computations according to a single instruction, multiple data ("SIMD") paradigm. In one or more implementations, APD 116 is used to implement a GPU, and in these implementations, the parallel processing units are referred to as shader cores or streaming multiprocessors (SMXs). Each parallel processing unit includes one or more processing elements, such as scalar and / or vector floating point units, arithmetic and logic units (ALUs), and the like. In various implementations, the parallel processing units also include special-purpose processing units (not shown), such as an inverse square root unit and a sine / cosine unit.

[0048] Although various functions are described herein as being performed by or in conjunction with APD 116, in various alternatives, the functions described as being performed by APD 116 are additionally or alternatively performed by other computing devices with similar capabilities that are not driven by a host processor (e.g., processor 102) and provide graphical output to display device 118. For example, any processing system that performs processing tasks according to the SIMD paradigm may perform the functions described herein. Alternatively, computing systems that do not perform processing tasks according to the SIMD paradigm may also perform the functions described herein.

[0049] Figure 21 is a block diagram of device 100 illustrating additional details related to executing processing tasks on APD 116. In one or more implementations, processor 102 maintains one or more control logic modules in system memory 104 for execution by processor 102. The control logic modules include, for example, operating system 202, kernel mode drivers 204, and applications 206. These control logic modules control various features of the operation of processor 102 and APD 116. For example, operating system 202 communicates directly with hardware and provides an interface to the hardware for other software executing on processor 102. Kernel mode drivers 204 control the operation of APD 116 by, for example, providing an application programming interface ("API") to software executing on processor 102 (e.g., application 206) to access various functions of APD 116. In one or more implementations, kernel mode drivers 204 also include a just-in-time compiler that compiles programs for execution by processing components of APD 116 (such as SIMD unit 210).

[0050] In one or more specific implementations, APD 116 includes any cooperative collection of hardware and / or software that performs functions and computations associated with accelerated graphics processing tasks, data parallel tasks, nested data parallel tasks in an accelerated manner relative to resources such as conventional CPUs, conventional GPUs, and combinations thereof. Although GPU, accelerated processing unit (APU), and general purpose graphics processing unit (GPGPU) are terms commonly used in the art, the expression "accelerated processing device (APD)" as used herein is a broader expression.

[0051] The APD 116 executes commands and programs for selected functions, such as graphics operations and non-graphics operations that may be suitable for parallel processing. In one or more specific implementations, the APD 116 is used to perform graphics pipeline operations, such as pixel operations, geometric calculations, and rendering images to a display device 118 based on commands received from the processor 102. The APD 116 also performs computational processing operations that are not directly related to graphics operations, such as operations related to video, physical simulations, computational fluid dynamics, or other tasks, based on commands received from the processor 102. For example, such commands include special instructions that are not generally defined in the instruction set architecture (ISA) of the APD 116. In at least some specific implementations, the APD 116 receives an image geometry representing a graphics image, and one or more commands or instructions for rendering and displaying the image. In various specific implementations, the image geometry corresponds to a representation of a two-dimensional (2D) or three-dimensional (3D) computerized graphics image.

[0052] APD 116 includes computing units 208 (illustrated as computing units 208-1 to 208-3), such as processing cores, which include one or more SIMD units 210 (illustrated as SIMD units 210-1 to 210-6) that perform operations in parallel at the request of processor 102 according to the SIMD paradigm. The SIMD paradigm is such a paradigm in which multiple processing elements share a single program control flow unit and program counter, and therefore execute the same program, but can execute the program with different data. In one example, each SIMD unit 210 includes a given number of channels, wherein each channel executes the same instruction simultaneously with other channels in the SIMD unit 210, but can execute the instruction with different data. If not all channels need to execute a given instruction, the channel can be closed by prediction. Prediction can also be used to execute programs with divergent control flow. More specifically, for programs with conditional branches or other instructions where control flow is based on computations performed by a single pass, passes corresponding to control flow paths that are not currently being executed are predicted, and serial execution of different control flow paths can achieve arbitrary control flow.

[0053] The basic execution unit in the computing unit 208 is a work item. Each work item represents a single instantiation of a program to be executed in parallel in a specific channel. Work items can be executed simultaneously as a "wavefront" on a single SIMD processing unit 210. One or more wavefronts are included in a "work group", which includes a collection of work items designated to execute the same program. The work group can be executed by executing each of the wavefronts that constitute the work group. In an alternative, the wavefront is executed sequentially on a single SIMD unit 210, or partially or completely in parallel on different SIMD units 210. The wavefront represents the maximum set of work items that can be executed simultaneously on a single SIMD unit 210. Therefore, if the command received from the processor 102 indicates that a specific program will be parallelized to the extent that the program cannot be executed simultaneously on a single SIMD unit 210, the program is divided into wavefronts that are parallelized on two or more SIMD units 210 or serialized on the same SIMD unit 210 (or parallelized and serialized as needed). Scheduler 212 performs operations related to scheduling the various wavefronts across different compute units 208 and SIMD units 210 .

[0054] The parallelism provided by one or more computing units 208 is suitable for graphics-related operations, such as pixel value calculations, vertex transformations, tessellation, geometric shading operations, and other graphics operations. Therefore, in one or more specific implementations, the graphics processing pipeline 214 that accepts graphics processing commands from the processor 102 provides computing tasks to the computing units 208 for parallel execution. Some graphics pipeline operations (such as pixel processing and other parallel computing operations) require the execution of the same command stream or computing kernel on a stream or set of input data elements. Corresponding instantiations of the same computing kernel are executed concurrently on multiple SIMD 210 units in one or more computing units 208 to process such data elements in parallel. As mentioned herein, for example, a computing kernel is a function that contains instructions declared in a program and executed on an APD computing unit 208. This function is also called a kernel, a shader, a shader program, or a program.

[0055] The graphics processing pipeline 214 includes multiple stages (not shown for simplicity) configured to process different primitives simultaneously in response to a draw call. Figure 2 206 (such as a video game). In one or more specific implementations, the stages of the graphics processing pipeline 214 simultaneously process different primitives generated by an application 206 (such as a video game). When geometry data is submitted to the graphics processing pipeline 214, hardware state settings are selected to define the state of the graphics processing pipeline 128. Examples of states include rasterizer states, blending states, depth stencil states, primitive topology types of the submitted geometry, and shaders (e.g., vertex shaders, domain shaders, geometry shaders, hull shaders, pixel shaders, etc.) used to render the scene. The shaders implemented in the graphics pipeline state are represented by corresponding byte codes. In some cases, information representing the graphics pipeline state is hashed or compressed to provide a more efficient representation of the graphics pipeline state.

[0056] The APD 116 is configured to implement features of the present disclosure by performing a number of functions as described in more detail below. For example, the APD 116 is configured to implement a super-resolution upscaler 332 that receives a low-resolution rendered frame 502 of a video stream. The super-resolution upscaler 332 spatially upscales the low-resolution rendered frame 502 by using temporal feedback (e.g., a previously upscaled frame of the video stream) to reconstruct a high-resolution frame 508 representing the rendered frame.

[0057] Figure 3 The invention illustrates a method according to one or more specific implementations. Figure 2. As described below, the graphics pipeline 214 can use temporal feedback (e.g., data associated with one or more previously upscaled frames) to perform super-resolution upscaling to reconstruct a high-resolution image while maintaining or improving image quality compared to native rendering. In at least some implementations, the graphics pipeline 214 is configured to perform super-resolution upscaling in order to reconstruct a high-resolution image. Figure 1 and Figure 2 The APD 116 is shown as being implemented in FIG. 1. It should be understood that the super-resolution upscaling techniques described herein are also applicable to other pipeline configurations.

[0058] In one or more specific implementations, the graphics processing pipeline 214 is configured to render graphics as images depicting a scene having three-dimensional geometry, but potentially two-dimensional geometry, in a virtual space (also referred to herein as "world space"). The graphics processing pipeline 214 typically receives a representation of a three-dimensional scene, processes the representation, and outputs a two-dimensional raster image. These stages of the graphics processing pipeline 214 process data that is initially properties at the endpoints (or vertices) of geometric primitives, where the primitives provide information about the object being rendered. Typical primitives in three-dimensional graphics include triangles and lines, where the vertices of these geometric primitives provide information about, for example, xyz coordinates, textures, and reflectivity.

[0059] Graphics pipeline 214 has access to storage resources 334 (also referred to herein as storage components), such as a hierarchy of one or more memories or caches that implement buffers and store vertex data, texture data, etc. For example, using Figure 1 and Figure 2 Some specific implementations of the system memory 104 shown implement storage resources 334. Some specific implementations of storage resources 334 include (or may access) one or more caches 336, one or more random access memory units 338, a video random access memory unit (not shown), one or more processor registers (not shown), etc., depending on the nature of the data at a particular stage in rendering. Therefore, it should be understood that storage resources 334 refer to any processor-accessible memory utilized in the graphics processing pipeline 214.

[0060] The graphics processing pipeline 214 includes stages that each perform a specific function. The stages represent a subdivision of the functionality of the graphics processing pipeline 214. Each stage is implemented in part or in whole as a programmable processing unit (such as Figure 2The graphics processing pipeline 214 may be implemented as a shader program executed in a SIMD unit 210 of the graphics processing unit, or partially or entirely as fixed-function non-programmable hardware external to the programmable processing unit. Stages 301 and 303 represent the front-end geometry processing portion of the graphics processing pipeline 214 before rasterization. Stages 305 to 311 represent the back-end pixel processing portion of the graphics processing pipeline 214.

[0061] In the input assembler stage 301 of the graphics pipeline 214, the input assembler 302 is configured to access information for objects defining portions of a model representing a scene from the storage resource 334. For example, in various implementations, the input assembler stage 220 reads primitive data (e.g., points, lines, and / or triangles) from a user-filled buffer (e.g., a buffer filled at the request of software executed by the processor 102, such as the application 206) and assembles such data into primitives to be used by other pipeline stages of the graphics processing pipeline 200. As used herein, the term "user" refers to an application 206 or other entity that provides shader code and three-dimensional objects for rendering to the graphics processing pipeline 214. The term "user" is used to distinguish activities performed by the APD 116. Input assembler 302 assembles vertices into several different primitive types (such as line lists, triangle strips, or primitives with adjacency) based on the primitive data included in the user fill buffer, and formats the assembled primitives for use by the rest of the graphics processing pipeline 214 .

[0062] In one or more specific implementations, the graphics processing pipeline 214 operates on one or more virtual objects defined by a set of vertices established in world space and having a geometry defined relative to coordinates in the scene. For example, the input data utilized in the graphics processing pipeline 214 includes a polygonal mesh model of the scene geometry (the vertices of the scene geometry correspond to the primitives processed in the rendering pipeline according to aspects of the present disclosure), and the initial vertex geometry is established in the storage resource 334 during the application stage implemented by the CPU.

[0063] In the vertex processing stage 303 of the graphics pipeline 214, one or more vertex shaders 304 process the vertices of the primitives through the input assembler 302. For example, the vertex shader 304 receives a single vertex of the primitive as input and outputs a single vertex. The vertex shader stage 304 performs various vertex-by-vertex operations, such as transformation, skinning, deformation, and vertex-by-vertex lighting calculations (lighting). Transformation operations include various operations for transforming the coordinates of the vertex (e.g., XY coordinates and Z depth values). These operations include one or more of modeling transformations, view transformations, projection transformations, perspective segmentation, and viewport transformations. In this article, such transformations are considered to modify the coordinates or "position" of the vertex on which the transformation is performed. Other operations of the vertex shader stage 304 modify other attributes except the coordinates.

[0064] In one or more specific implementations, the vertex shader stage 304 is partially or completely implemented as a vertex shader program to be executed on one or more computing units 208. The vertex shader program is provided by the processor 102 and is based on a pre-written program by a computer programmer. The kernel mode driver 204 compiles such a computer program to generate a vertex shader program, which has a format suitable for execution within the computing unit 208. Some specific implementations of shaders such as the vertex shader 304 implement large-scale single instruction multiple data (SIMD) processing, so that multiple vertices are processed simultaneously. In at least some specific implementations, the graphics pipeline 214 implements a unified shader model so that all shaders included in the graphics pipeline 214 have the same execution platform on a shared large-scale SIMD computing unit 210. In these specific implementations, the shaders including the vertex shader 304 are implemented using a common resource set referred to herein as a unified shader pool 306.

[0065] In one or more implementations, the vertex processing stage 303 also performs additional vertex processing calculations that subdivide the primitives and generate new vertices and new geometry in world space. In at least some implementations, these additional vertex processing calculations are performed by one or more of the hull shader 308, the tessellation device 310, the domain shader 312, and the geometry shader 314. The hull shader 308 operates on input high-order patches or control points that define input patches. The hull shader 308 outputs tessellation factors and other patch data. In one or more implementations, the primitives generated by the hull shader 308 are provided to the tessellation device 310. The tessellation device 310 receives an object (such as a patch) from the hull shader 308 and generates information identifying the primitives corresponding to the input object, for example, by tessellizing the input object based on the tessellation factors provided to the tessellation device 310 by the hull shader 308. Tessellation subdivides input high-order primitives (such as patches) into a set of low-order output primitives that represent a finer level of detail, e.g., as indicated by a tessellation factor that specifies the granularity of the primitives produced by the tessellation process. Thus, a model of the scene is represented by a smaller number of higher-order primitives (to save memory or bandwidth), and additional detail is added by tessellizing the higher-order primitives.

[0066] The domain shader 312 inputs the domain orientation and (optionally) other patch data. The domain shader 312 operates on the information provided and generates a single vertex for output based on the input domain orientation and other information. The geometry shader 314 receives the input primitives and outputs up to four primitives generated by the geometry shader 314 based on the input primitives. In some specific implementations, the geometry shader 314 retrieves vertex data from the storage resource 334 and generates new graphics primitives, such as lines and triangles, from the vertex data in the storage resource 334. Specifically, the shader 314 retrieves vertex data for the primitives as a whole and generates zero or more primitives. For example, the geometry shader 314 can operate on a triangle primitive with three vertices. A variety of different types of operations can be performed by the geometry shader 314, including operations such as: point sprite expansion, dynamic particle system operations, fur-fin generation, shadow volume generation, single-pass rendering to a cube map, per-primitive material exchange, and per-primitive material establishment. In at least some implementations, one or more of hull shader 308 , domain shader 312 , or geometry shader 314 are implemented as shader programs to be executed on a programmable processing unit such as SIMD unit 210 , while tessellator 310 is implemented by fixed function hardware.

[0067] Once front-end processing is complete, the scene is defined by a set of vertices, each vertex having a set of vertex parameter values ​​stored in storage resources 334. In some implementations, the vertex parameter values ​​output from vertex processing stage 303 include positions defined by different homogeneous coordinates for different regions.

[0068] As described above, stages 305 to 311 represent the back-end processing of the graphics processing pipeline 214. The rasterizer stage 305 includes a rasterizer 316 that accepts simple primitives generated upstream and rasterizes these primitives. The rasterizer 316 performs shading operations and other operations such as clipping, perspective division, shearing, and viewport selection. The rasterizer 316 generates a set of pixels that are subsequently processed in the pixel processing / shader stage 307 of the graphics pipeline 214. In some specific implementations, the pixel set includes one or more tiles. In one or more specific implementations, the rasterizer 316 is implemented by fixed function hardware.

[0069] The pixel processing stage 307 includes one or more pixel shaders 318, which input a pixel stream (e.g., including a set of pixels generated by a rasterizer 316) and output zero or another pixel stream in response to the input pixel stream. Based on the primitives generated upstream and the results of rasterization, the pixel shader 318 calculates output values ​​for screen pixels. In one or more specific implementations, the pixel shader 318 applies a texture from a texture memory, which can be implemented as part of the storage resource 334. The outputs (such as color values, depth values, and stencil values) generated by the one or more pixel shaders 318 are stored in one or more corresponding buffers (such as a color buffer 320, a depth buffer 322, and a stencil buffer 324), respectively. The combination of the color buffer 320, the depth buffer 322, and (optionally) the stencil buffer 324 is referred to as a frame buffer 326. In at least some specific implementations, the graphics pipeline 214 implements multiple frame buffers 326, including a front buffer, a back buffer, and an intermediate buffer, such as a render target, a frame buffer object, etc. The operations of pixel shader 318 are performed by a shader program executing on programmable processing unit 210 .

[0070] In one or more specific implementations, the pixel shader 318 or another shader accesses shader data, such as texture data, stored in the storage resource 334. The texture data defines textures, which are bitmap images used at various points in the graphics processing pipeline 214. For example, in some examples, the pixel shader 318 applies textures to pixels to increase apparent rendering complexity (e.g., to provide a more "realistic" appearance) without increasing the number of vertices to be rendered. In another instance, the vertex shader 304 uses texture data to modify primitives to increase complexity and thereby improve aesthetics by, for example, creating or modifying vertices. For example, the vertex shader stage 304 uses a height map stored in the storage resource 334 to modify the displacement of a vertex. This type of technique can be used to generate more realistic-looking water, for example, by modifying the position and number of vertices used to render water compared to using only textures for the pixel processing stage 307. In some instances, the geometry shader 314 also accesses texture data from the storage resource 334.

[0071] The output merger stage 309 (including the output merger 328, which accepts the outputs from the pixel processing stage 307 and merges those outputs) performs operations such as z-testing, alpha blending, stencil printing, or other operations on each pixel received from the pixel shader 318 to determine the final color for the screen pixel. For example, the output merger 328 combines various types of output data (e.g., pixel shader values, depth and stencil information, etc.) with the contents of the color buffer 320, the depth buffer 322, and (optionally) the stencil buffer 324, and stores the combined output back into the frame buffer 326. The output of the merger stage 309 can be referred to as rendered pixels that collectively form a rendered frame. In one or more specific implementations, the output merger 328 is implemented by fixed function hardware.

[0072] A post-processing stage 311 is implemented after the output merger stage 309. In the post-processing stage 311, one or more post-processors 330 operate on the rendered frame (or individual pixels) stored in the frame buffer 326 to apply one or more post-processing effects, such as ambient occlusion or tone mapping, before outputting the frame to a display. In at least some implementations, the post-processor 330 is implemented using one or more vertex shaders 304, one or more pixel shaders 318, and the like. The post-processed frame is written to a frame buffer 326, such as a back buffer for display or an intermediate buffer for further post-processing. In at least some implementations, the graphics pipeline 214 includes other shaders or components that can communicate with one or more of the other components of the graphics pipeline 214, such as a computer shader 340, a ray tracer 342, a mesh shader 344, and the like. At least a portion of post-processing stage 311 , vertex processing stage 303 , rasterization stage 305 , and pixel processing stage 307 are collectively referred to herein as a “renderer 313 ” or “rendering stage 313 ” of graphics pipeline 214 .

[0073] In many instances, the amount of processing resources necessary to render a full high-resolution image may make it difficult to render frames while meeting current frame rates, such as at least sixty frames per second (fps). Therefore, in one or more implementations, an application 206 (e.g., a video game) uses a renderer 313 to generate rendered images at a rendering resolution or size (e.g., 1920×1080 pixels) that is lower than one or more final output / rendering resolutions or sizes (e.g., 3840×2160 pixels). To meet timing requirements and reduce processing resource requirements. In at least some implementations, a super-resolution upscaling stage 315 including a super-resolution upscaling stage 332 (referred to herein as “upscaling stage 332” for simplicity) processes the low-resolution rendered image to generate an upscaled image representing the contents of the low-resolution rendered image at a resolution equal to (or at least closer to) the target rendering resolution. In one or more implementations, the upscaling stage 315 is implemented as part of the post-processing stage 311. However, in other implementations, the upscaling stage 332 is part of a different processing stage of the graphics pipeline 214. In one or more implementations, the amplifier 332 is implemented using one or more vertex shaders 304, pixel shaders 318, compute shaders 340, etc., or a combination thereof.

[0074] The amplifier 332 improves application rendering performance by implementing a time magnification algorithm that operates on several inputs. Therefore, in one or more specific implementations, the amplifier 332 is positioned in the graphics pipeline 214 to ensure the balance between the highest quality visual quality and performance. Placing other image space processing before the amplifier has the following advantages: these other image space processing runs at a lower resolution, which provides performance advantages for the application 206. However, this placement may not be suitable for certain categories of image space processing / techniques. For example, some image space processing may introduce noise or particles into the final image (for example, to simulate a physical camera). Placing these image space processing before the amplifier may cause the amplifier to amplify the noise, which leads to undesirable artifacts in the resulting magnified image. Therefore, in one or more specific implementations, the amplifier 332 is placed between the post-processor 330 that usually operates on the current frame at the rendering resolution and the post-processor 330 that operates on the magnified frame generated by the amplifier 332 at the rendering resolution. However, in at least some specific implementations, techniques such as noise and particles can be calculated at a low resolution and applied to / synthesized to the magnified frame using, for example, bilinear sampling.

[0075] Figure 4 A graphics pipeline integration diagram 400 for an upscaling 332 is illustrated. In this example, the first stage is the rendering portion 313 of the graphics pipeline 214, which renders a current frame in a rendering color space at a given rendering resolution. The pre-amplification post-processing stage 311-1 takes the rendered frame as input and performs one or more post-processing operations on the rendered frame at the rendering resolution and in the rendering color space. The pre-amplification post-processing stage 311-1 includes a post-processor 330 that performs post-processing operations, which typically access the depth buffer 322 of the rendered frame. Examples of these post-processing operations include screen space reflections, screen space ambient occlusion, denoising (e.g., shadows or reflections), exposure, etc. The upscaling stage 315 takes the output of the pre-amplification post-processing stage 311-1 (or the rendering stage 313) as input and performs at least one of an upscaling operation or an anti-aliasing operation on the pre-scaled post-processed frame or the rendered frame (if post-processing has not yet been performed) in a linear color space. The output of the enlargement stage 315 is at least one of an enlarged frame or an anti-aliased frame having a target presentation resolution in a presentation color space. The post-enlargement post-processing stage 311-2 takes the enlarged (and anti-aliased) frame as input and performs one or more post-processing operations on the enlarged frame at the presentation resolution and in the presentation color space. The post-enlargement post-processing stage 311-2 includes post-processors 330 that perform operations that typically use the anti-aliased frame or add effects to the frame that would introduce undesirable artifacts when the frame is enlarged. Examples of these post-processing operations include film grain, chromatic aberration, vignetting, tone mapping, halo, depth of field, motion blur, etc. Figure 4The final stage 417 in the example shown is the rendering of the user interface or heads-up display, which the graphics pipeline 214 renders at the presentation resolution and in the presentation color space.

[0076] Figure 5 is a block diagram illustrating a more detailed view of the amplifier 332. As described below, the amplifier 332 is temporal and takes as input data associated with a currently rendered (aliased) frame 502 (also referred to herein as the "current frame 502" or "input frame 502") and data associated with a previously rendered (enlarged) frame. In one or more specific implementations, the rendered frame 502 and the previously enlarged frame are frames of a video stream. The previously rendered frame is the most recent frame processed by the amplifier 332 and presented to the user at the target rendering resolution. In one or more specific implementations, the input data associated with the previously rendered frame includes an output buffer 506-1 that holds data for the previously rendered frame, such as a color buffer (e.g., output color), a depth buffer, etc. The resolution / size of the output buffer 506-1 corresponds to the rendering resolution.

[0077] In one or more implementations, the input data associated with the current frame 502 includes various buffers provided by the application 206, such as the color buffer 320-1, the depth buffer 322-1, and the motion vector buffer 504. The input color buffer 320-1 is a rendering resolution color buffer for the current frame 502 provided by the application 206. In one or more implementations, the input color buffer 320-1 includes color data, such as pixel color values, generated based on sub-pixel dithering performed during rendering of the current frame 502. Sub-pixel dithering can be performed for each frame in the frame sequence during the rendering phase. During sub-pixel dithering, the center point of the color determination for the pixel of the frame is slightly shifted to another point in the pixel using a sub-pixel offset, and the color data is determined for this offset position. In one or more implementations, the output merger 328 of the graphics pipeline 214 combines the multi-sample color data determined for each sub-pixel sample to determine the final color data for the corresponding pixel. The output merger 328 then stores the final color data in the color buffer 502-1 of the frame 320. In one or more implementations, the dithering orientation is determined randomly or according to a determined pattern or sequence, such as a Halton sequence (e.g., Halton [2,3]). The Halton sequence provides spatially separated points that cover the available pixel space. In one or more implementations, dithering is applied to the rendering of multiple object types including opaque, alpha transparent, and ray traced objects. For rasterized objects, sub-pixel dithering values ​​can be applied to the camera projection matrix, which is then used to perform transformations during vertex shading. For ray tracing rendering, sub-pixel dithering is applied to the origin of the ray source, which is typically the location of the camera.

[0078] In some configurations, the input depth buffer 322-1 includes the stencil buffer 324. However, in other configurations, the input depth buffer 322-1 is stored separately from the stencil buffer 324. The input depth buffer 322-1 is a rendering resolution depth buffer for the current frame 502 provided by the application 206. In one or more specific implementations, the resolution / size of the input color buffer 320-1 and the input depth buffer 322-1 is equal to the rendering resolution. The motion vector buffer 504 includes 2D motion vectors that encode the motion from a pixel in the current frame 502 to the position of the same pixel in the previous (zoomed-in) frame. In one or more specific implementations, the motion vectors are provided by the application 206 in a range of, for example, [(<-width, -height>..<width, height>)]. For example, a motion vector for a pixel in the upper left corner of the screen with a value of <width, height> represents a motion across the entire width and height of the input surface starting from the lower right corner of the screen. In at least some configurations, one or more of the opaque, alpha-tested, or alpha-blended objects writes its motion vector for all covered pixels. If vertex shader effects are applied, such as scrolling UVs, these calculations are decomposed into motion calculations that enhance the results. In some configurations, the resolution / size of the motion vector buffer 504 is equal to the rendering resolution. However, in other configurations, the resolution / size of the motion vector buffer 504 is equal to the presentation resolution. In one or more specific implementations, the color buffer 320-1, the depth buffer 322-1, and the motion vector buffer 504 are texture data types, but other data types may also be applicable.

[0079] In at least some implementations, when magnifying the rendered frame 502, the magnifier 332 processes additional external input resources, such as the reactive mask 602 ( Figure 6 ) and exposure 604( Figure 6 In the context of the amplifier 332, "reactivity" refers to how much influence the samples rendered for the current frame 502 have on the generation of the final upscaled image. In one or more implementations, the samples rendered for the current frame 502 contribute relatively little to the results computed by the amplifier 332, but there are exceptions. For example, to produce the best results for fast-moving alpha-blended objects, one or more stages of the amplifier 332 (e.g., the reprojection and accumulation stages 611 ( Figure 6)) is configured to be able to become more reactive to such pixels. Since it may be difficult to determine which pixels have been rendered using alpha blending based on color, depth, or motion vectors, in one or more specific implementations, the application 206 provides a reactive mask 602 as an input to the amplifier 332. The reactive mask 602 provides a mechanism for the application 206 to identify areas of the rendered frame 502 that do not leave an occupied area or include a motion vector 504 in the input depth buffer 322-1. In other words, the reactive mask 602 guides the amplifier 332 where it should reduce the amplifier's reliance on historical information when synthesizing the current pixel, and instead allow samples of the current frame 502 to contribute more to the final result. The reactive mask 602 allows the application 206 to provide a value from, for example, [0...1], where a value of 0 indicates that the pixel is not reactive at all (e.g., the amplifier 332 will use its default synthesis strategy), and a value of 1 indicates that the pixel is fully reactive. In one or more specific implementations, the alpha value used when synthesizing an alpha-blended object (e.g., a particle) into a scene is implemented as a proxy for reactivity. In these implementations, the application 206 writes the alpha value for each covered pixel of the alpha-blended object to the corresponding pixel in the reactive mask 602. In one or more implementations, the resolution / size of the reactive mask 602 is equal to the rendering resolution of the rendering resolution. Furthermore, in one or more implementations, the reactive mask 602 is a texture data type, although other data types may also be applicable.

[0080] Exposure 604 informs amplifier 332 of the exposure value calculated by application 206 for rendered frame 502. In one or more implementations, the exposure value matches the exposure used by application 206 during any subsequent tone mapping pass. In at least some implementations, amplifier 332 calculates its own exposure value at various stages for internal use. In addition, in one or more implementations, the output generated by amplifier 332 inverts this internal tone mapping before the final output is written. In other words, amplifier 332 returns a result that is in the same domain (or close to the same domain) as the original input signal. In one or more implementations, exposure 604 is a texture data type, but other data types may also be applicable.

[0081] The amplifier 332 uses the current frame input and the previous frame input to generate a super-resolution upscaled (and anti-aliased) frame 508 (also referred to herein as "upscaled frame 508" for simplicity) corresponding to the rendered frame 502 at the target rendering resolution. In one or more specific implementations, the amplifier 332 stores data representing the upscaled frame 508 in the output buffer 506-2. The amplifier 332 maintains an output buffer history 510 for accessing the output buffer 506-1 generated for the previously rendered frame. The amplifier 332 implements one or more components to perform the upscaling operations and anti-aliasing operations described herein. For example, the amplifier 332 implements an automatic exposure component 512, an input color adjustment component 514, a reconstruction and dilation component 516, a depth clipping component 518, a locking component 520, a reprojection and accumulation component 522, and a sharpening component 524. Each of these components is implemented in a corresponding stage of the amplifier 332, as described below with respect to Figure 6 It should be understood that although the components 512 to 524 of the amplifier 332 are illustrated as being implemented separately from one another, two or more of these components may be combined.

[0082] Figure 6 Various stages / passes of the enlarger 332 are illustrated, such as an auto-exposure stage 601, an input color adjustment stage 603, a reconstruction and dilation stage 605, a depth cropping stage 607, a locking stage 609, a reprojection and accumulation stage 611, and a sharpening stage 613. Figure 6 The inputs processed by each stage and the outputs generated by each stage are further illustrated. Figure 6 In the example shown, shaded boxes represent stages of the amplifier 332, dashed boxes represent input / output buffers, and solid boxes represent intermediate / working buffers. In one or more implementations, the application 206 interacts with the amplifier 332 through one of a plurality of different application programming interfaces (APIs). For example, the application 206 can instantiate the amplifier 332 through the API, issue one or more calls to the amplifier 332, pass one or more data structures and inputs to the amplifier 332, etc. In one or more implementations, when the amplifier 332 is instantiated (e.g., called by the application 206), storage resources 334 (such as GPU local memory) are allocated for consumption by the amplifier 332. The amplifier 332 uses these storage resources 334 to store intermediate textures calculated by the amplifier 332, and also to store textures that persist across many frames of the application 206.

[0083] In the auto-exposure stage 601, the application 206 provides the color buffer 320-1 of the currently rendered frame 502 to the auto-exposure component 512 as input. If the contents of the color buffer 320-1 are in high dynamic range (HDR), the application 206 can indicate this to the auto-exposure component 512 by, for example, setting a flag in a data structure provided by the application 206 to the amplifier 332. The auto-exposure component 512 processes the color buffer 320-1 to generate up to two intermediate storage resources depending on the configuration provided by the application 206. The first intermediate storage resource is the current brightness texture 606, which is a lower resolution representation of the brightness of the input color. For example, the current brightness texture 606 is a texture at 50% (or other percentage) of the rendering resolution texture (e.g., the color buffer 320-1), which includes the brightness values ​​of the currently rendered frame 502. As described below, the current brightness texture 606 is used by the shading change detection process in the reprojection and accumulation stage 611. The second intermediate storage resource is an exposure texture 604 that includes the exposure value calculated for the rendered frame 502. The exposure texture 604 is optionally used by subsequent stages of the amplifier 332, depending on the configuration of the amplifier 332. For example, the application 206 can indicate whether the exposure 604 will be used during the upscaling process by, for example, setting a flag in a data structure provided to the amplifier 332. In other specific implementations, the exposure 604 is provided by the application 206 to the automatic exposure component 512 as input. The exposure 604 is optionally used by the exposure calculation of the input color adjustment stage 603 to apply tone mapping, and is used by the reprojection and accumulation stage 611 to invert the local tone mapping before generating an output (e.g., the upscaled frame 508) by the amplifier 332. In one or more specific implementations, the exposure 604 has a resolution / size of 1×1 pixels, but other sizes may also be applicable.

[0084] In at least some implementations, the auto-exposure component 512 implements at least one downsampling mechanism, such as AMD A single-pass downsampler, at least one downsampling mechanism uses a shader dispatch (such as a single compute shader dispatch) to generate a mipmap chain. A mipmap is a collection of bitmap images of textures with successively lower resolutions. In other words, a mipmap includes multiple versions of the same texture, each with a different resolution. These different versions may be referred to as "mipmap levels," "levels," or "mips." In one or more specific implementations, instead of a conventional pyramid approach, the downsampling mechanism implemented by the auto-exposure component 512 generates a specific set of mipmap levels for any input texture (e.g., the input color buffer 320-1), and performs any calculations on the data when storing the data in a target orientation in memory.

[0085] In one or more specific implementations, the downsampling mechanism of the auto-exposure component 512 is configured to write only to the second (eg, half-resolution) mipmap level 702 and the last (1×1) mipmap level 704, such as Figure 7 exemplified in . Furthermore, different calculations are applied at each of these levels 702 and 704 to calculate the amounts used by subsequent stages of the amplifier 332. Therefore, the remaining levels of the mipmap chain do not need to be backed up by GPU local memory (or any other type of memory). The second mipmap level 702 includes a current brightness 606, and the last mipmap level 704 includes an exposure 604, the value of which is calculated during the downsampling of the color buffer 320-1. In one example, the current brightness 606 includes a texture at 1 / 32 rendering resolution, so that each pixel includes the average brightness of a 32×32 pixel area in the source texture. The average brightness of the 32×32 pixel area surrounding the input pixel is obtained by using the coordinates of the current pixel and sampling into the texture using bilinear sampling.

[0086] Back to Figure 6In one or more implementations, the various stages of the amplifier 332 operate in a different color space (such as YCoCg) than the rendered frame 502 (e.g., RGBa). Therefore, to avoid repeatedly calculating conversions from the color space used by the application 206, the amplifier 332 implements an input color adjustment stage 603 that applies all adjustments to the color once. For example, in the input color adjustment stage 603, the input color adjustment component 514 takes as input the input color buffer 320-1 provided by the application 206 and the exposure 604 provided by the application 206 or generated by the auto-exposure component 512 in the auto-exposure stage 601. The input color adjustment component 514 processes these inputs to perform various adjustment operations on the input colors of the rendered frame 502. One example of these adjustment operations includes dividing the input color (e.g., RGB) by a pre-exposure value provided by the application 206 as part of the current frame 502 to obtain the original input color. In many cases, different frames use different pre-exposure values ​​(to aid in computational accuracy from a gaming perspective), and the original input color of the frame is divided by the pre-exposure value to obtain all images in a comparable range space. Another example of an adjustment operation includes aligning the input image to a mid-gray level by multiplying the input color by the exposure value 604. Another example of an adjustment operation includes converting the exposure color to a YCoCg color space. The YCoCg color space includes a luma value (Y) and two chroma values, namely green chroma (Cg) and orange chroma (Co). The luma value (Y) represents the brightness in the rendered frame 502 (the achromatic portion of the frame), and the chroma components (Co and Cg) represent the color information of the rendered frame 502. The results of the input color adjustment stage 603 are then cached to an adjusted color buffer (texture) 608, which can then be read by subsequent stages of the amplifier 332. In one or more specific implementations, the resolution / size of the adjusted color buffer 608 is equal to the rendering resolution.

[0087] As part of the adjustment process, the input color adjustment component 514 also generates a luma history buffer (texture) 610 having a resolution / size equal to the rendering resolution. The luma history buffer 610 includes the luma values ​​Y of the currently prepared input colors of the rendered frame 502. In one or more implementations, the luma history buffer 610 is persistent (i.e., it is not available for aliasing and is not cleared on each frame). Therefore, multiple frames (e.g., four frames) of luma history are maintained in the luma history buffer 610 and can be accessed for any frame during the input color adjustment stage 603. However, at the end of the input color adjustment stage 603, the luma history values ​​are shifted downward. In other words, in at least some implementations, subsequent stages of the amplifier 332 have access to a lower number of luma frames, e.g., the three most recent luma frames (the current frame and the two previous frames). Therefore, in this example, if the current frame is denoted as n, the values ​​stored in the luma history buffer 610 are as follows:

[0088]

[0089] Table 1

[0090] In one or more implementations, the input color adjustment component 514 encodes the stability factor into the alpha channel of the luma history buffer 610. The stability factor is a measure of the stability of luma over the current frame 502 and a given number of frames coming before the current frame 502 (e.g., three frames).

[0091] In addition to performing the input color adjustment operations described above, the input color adjustment component 514 clears the reprojected / previous depth buffer 616 to a known value as described below. This process prepares the previous depth buffer 616 for the reconstruction and dilation stage 605 on the next rendered frame of the application 206. The input color adjustment component 514 selects the clear value based on, for example, the configuration of the previous depth buffer 616. For example, the input color adjustment component 514 clears the previous depth buffer 616 to a maximum far section distance value (z-far value), which is typically 0 for inverted depth. In at least some configurations, the previous depth buffer 616 is cleared as a result of the reconstruction and dilation stage 605 using an atomic operation to fill the previous depth buffer 616.

[0092] In the reconstruction and dilation stage 605, the reconstruction and dilation component 516 takes as input the input depth buffer 322-1 and the motion vector buffer 504 provided by the application 206. The reconstruction and dilation component 516 processes this input to produce a dilated depth buffer 612 for the previous frame, a buffer 614 including a dilated set of motion vectors in UV space, and a reprojected / previous depth buffer 616. In one or more specific implementations, the reconstruction and dilation component 516 applies motion vector scaling to transform non-screen space motion vectors into screen space motion vectors before processing the motion vectors. The dilated depth buffer 612 is a texture that includes dilated depth values ​​determined from the input depth buffer 322-1. The dilated motion vector buffer 614 is a texture that includes dilated 2D motion vectors determined from the input motion vector buffer 504. In one or more specific implementations, the first color channel (such as the red channel) and the second color channel (such as the green channel) of the dilated motion vector buffer 614 include 2D motion vectors in normalized device coordinate (NDC) space. The previous depth buffer 616 is a texture that includes reconstructed depth values ​​of previous frames. Each of these buffers has a resolution / size equal to the rendering resolution of the current frame 502.

[0093] In more detail, the reconstruction and dilation component 516 calculates dilated depth values ​​and motion vectors based on input depth values ​​and motion vectors, which are included in the input depth buffer 322-1 and the input motion vector buffer 504 for the current frame 502, respectively. The dilated depth values ​​and motion vectors emphasize the edges of geometric structures that have been rendered into the input depth buffer 322-1. For example, the edges of geometric structures often introduce discontinuities into a continuous series of depth values. Therefore, as the depth values ​​and motion vectors are dilated, they naturally follow the contours of the geometric edges present in the input depth buffer 322-1. In one or more specific implementations, the reconstruction and dilation component 516 calculates the dilated depth values ​​and motion vectors by considering the depth values ​​of a 3×3 (or other kernel size) neighborhood for each pixel of the current frame 502. The reconstruction and dilation component 516 then selects the depth value and the corresponding motion vector in the neighborhood whose depth value is closest to the camera. This process is performed in Figure 8 Example in Figure 8 An example of the geometry 802 of the current frame 502 is shown in which the center pixel 804 of the 3×3 kernel 806 is updated with the depth value and motion vector from the pixel 808 having the maximum depth value. The reconstruction and dilation component 516 stores the determined dilated depth value in the dilated depth buffer 612 and the determined dilated motion vector in the dilated motion vector buffer 614.

[0094] The reconstruction and expansion component 516 uses the expanded motion vector of the expanded motion vector buffer 614 to estimate the position of each pixel from the depth buffer 322-1 of the current frame in the previous frame. For example, the reconstruction and expansion component 516 applies the expanded motion vector calculated for the pixel to the value of the pixel in the expanded depth buffer 612 to determine the position of the pixel in the previous frame. In other words, each depth sample from the expanded depth buffer 612 is reprojected to the position of the depth sample in the previous frame using the expanded motion vector of the sample. The reprojected depth samples are scattered among the affected depth samples using, for example, backward or reverse reprojection. Since many pixels are likely to be reprojected into the same pixel in the previous frame, the reconstruction and expansion component 516 uses atomic operations to resolve the value of the most recent depth value for each pixel. For example, in one or more specific implementations, the reconstruction and expansion component 516 uses atomic operations, such as InterlockedMax or InterlockedMin provided by a high-level shader language (HLSL), or comparable equivalents. In some implementations, the reconstruction and dilation component 516 performs different atomic operations (eg, InterlockedMax or InterlockedMin) depending on whether the input depth buffer 322 - 1 is inverted or non-inverted. The reconstruction and dilation component 516 stores the reconstructed / estimated depth values ​​in a reconstructed / previous depth buffer 616 .

[0095] Return to Figure 6 In the depth clipping stage 607, the depth clipping component 518 takes as input the dilated depth buffer 612, the dilated motion vector buffer 614, and the previous depth buffer 616. The depth clipping component 518 processes this input to generate a deocclusion mask / map 618 indicating the deoccluded areas of the current rendered frame 502. For example, when the camera moves from an initial position (previous frame) to a new position (current frame), pixels that were initially occluded from the viewpoint of the camera's previous position may become visible (deoccluded) from the viewpoint of the camera's current position. In one or more specific implementations, the deocclusion mask 618 is a texture that includes values ​​indicating how much the corresponding pixel of the current frame 502 has been deoccluded. In one example, a value of 0 indicates that the pixel was completely occluded in the previous frame and is now deoccluded, and a value of 1 indicates that the pixel was completely visible in the previous frame and is completely visible in the current frame 502. Values ​​between 0 and 1 indicate that the pixel was visible in the previous frame to an extent proportional to the value. In one or more implementations, the de-occlusion mask 618 has a resolution / size equal to the rendering resolution.

[0096] The depth clipping component 518 generates a deocclusion mask 618 by, for example, calculating a depth value for each pixel based on the position of the previous camera and the position of the new camera. The depth clipping component 518 compares the delta between the depth values ​​with a separation value (such as an Akeley separation constant k). sep ) is compared. The Akeley separation constant provides a minimum distance between two objects represented in the floating-point depth buffer, which the depth clipping component 518 uses to determine with a high degree of certainty that the pixels were originally different from each other. In one or more specific implementations, if the depth clipping component 518 determines that the delta between the depth values ​​is greater than the separation value calculated for the applied depth buffer 612 configuration, the depth clipping component 518 determines that the pixels represent different objects. However, if the depth clipping component 518 determines that the delta between the depth values ​​does not exceed the separation value, the depth clipping component 518 cannot determine with certainty that the pixels represent different objects. The depth clipping component 518 stores values ​​for pixels in the range of, for example, [0...1] in the de-occlusion mask 618, with a value of 1 mapping to a delta greater than or equal to the separation value.

[0097] In the lock stage 609 of the amplifier 332, the lock component 520 creates a new pixel lock based on the pixels of the current frame 502. As described below, the pixel lock is consumed in the reprojection and accumulation stage 611 of the amplifier 332. The lock component 520 takes as input the input lock state information 620, such as the input lock state texture / buffer 620-1 (also referred to herein as "lock state 620-1" for simplicity) and the adjusted color buffer 608. Based on this input, the lock component 520 generates a lock for one or more pixels of the current frame 502, if applicable. In one or more specific implementations, the generated lock is stored in the output lock state texture / buffer 620-2 (also referred to herein as "lock state 620-2" for simplicity). In one or more specific implementations, the amplifier 332 maintains the lock state 620 as an array of two textures (i.e., the input lock state 620-1 and the output lock state 620-2). Input lock state 620-1 is a lock state texture used as input by amplifier 332 when processing current frame 502, and is generated for a previously enlarged frame. Output lock state 620-2 is a lock state texture generated for current frame 502, which is used as input for the next frame to be processed by amplifier 332. Input lock state 620-1 and output lock state 620-2 each have a resolution / size equal to the target (enlarged) rendering resolution to maintain lock state 620. The two texture array configurations of lock state 620 allow reading of adjacent pixels of pixels being processed by lock stage 609 while avoiding read-modify-write conflicts. It should be understood that lock stage 609 is not limited to image enlargement. For example, lock stage 609 is also applicable to temporal anti-aliasing (TAA) and other techniques.

[0098] In at least some implementations, the input lock state 620-1 and the output lock state 620-2 each include lock state information for each pixel of the frame to which it is associated, the lock state information indicating whether the pixel is associated with a lock or unlocked. In one or more implementations, the lock acts as a classifier that indicates that the locked pixel includes high frequency information (e.g., brightness variations between relative neighbors of the pixel). In at least some implementations, the lock also acts as a mask that indicates that color correction in the reprojection and accumulation stage 611 is not performed on the pixel associated with the lock. The net effect of the pixel being locked is that more color data of the previous frame is used when calculating the final super-resolution pixel color during the reprojection and accumulation stage 611. For example, if the coloring changes between frames, the subsequent color correction process may limit the influence of the historical color so that the historical color is close to the color around the sample in the current frame 502, which causes small / fine features that are not visible in each dithering rendering to be clamped and disappear in the blended image. The lock increases the contribution of samples that include small / fine features, which prevents these features from being clamped and disappearing in the blended image.

[0099] In one or more specific implementations, the lock state 620 consists of two values, namely a red channel value and a green channel value. In other specific implementations, the lock state 620 consists of three values ​​(red channel, green channel and blue channel). The red channel of the lock state 620 includes the remaining life of the pixel lock and is initially set based on the jitter sequence length. For example, if 2× amplification is performed, the sequence length is 32 and the lock is potentially valid for 32 frames. The length of 32 frames is stored in the red channel as the initial life / life of the lock, and (in the 2× amplification example) each frame is reduced by a factor of approximately 1 / 32. In one or more other specific implementations, when the pixel is locked, the lock component 520 fills the red channel of the output lock state 620-2 with the remaining life value, which indicates whether the lock is new (i.e., created for the current frame) and whether the lock is the first / initial lock or a subsequent lock (e.g., the second lock) for the pixel. For example, the lock component 520 fills the red channel of the output lock state 620-2 with a remaining lifetime value of -1 to indicate that the lock is the first lock for the pixel, where the negative sign indicates that the lock is a new lock. If the pixel was locked in the previous frame, the new lock is considered to be a subsequent pixel lock (e.g., a second lock), and the lock component 520 fills the red channel of the output lock state 620-2 with a remaining lifetime value of -2 to indicate that the lock is new (i.e., generated for the current frame) and a subsequent lock for the pixel. In one or more specific implementations, if the remaining lifetime of the lock has an initial value of -1, the remaining lifetime is degraded between 0 and 1, and if the remaining lifetime of the lock has an initial value of -2, the remaining lifetime is degraded between 0 and 2. As described in more detail below, in these specific implementations, the remaining lifetime is dynamically degraded based on the contribution of the locked pixel to the final color for the current frame, rather than reducing the remaining lifetime by a factor of 1 / 32 (in the 2× magnification example). In addition, when the final lock for the pixel is stored during the reprojection and accumulation stage 611, the absolute value of the remaining lifetime is stored. Thus, in at least some implementations, the red channel of the final output lock state 620-2 includes a non-negative value for the remaining lifetime. Thus, when the final output lock state 620-2 becomes the input lock state 620-1 for the next frame, the remaining lifetime stored in the red channel is also a non-negative value.

[0100] The green channel of the lock state 620 includes the current brightness of the current frame 502 when the pixel is locked (e.g., the average brightness of the scene in the area around the pixel), and is obtained by the lock state 620 from the brightness texture 606. In at least some specific implementations, the lock component 520 populates the green channel of the lock state 620 during the reprojection phase of the reprojection and accumulation phase 611. When processing the next frame, the brightness information stored in the green channel of the currently generated lock state 620 is used by the reprojection and accumulation phase 611. The brightness value stored in the green channel of the lock state 620 is used as part of the color change detection for the current frame 502 in the reprojection and accumulation phase 611, which allows the amplifier 332 to unlock the pixel if there is a discontinuous change in the appearance of the pixel (e.g., a sudden change in the color of the pixel). In at least some specific implementations, the blue channel of the lock state 620 is populated with a confidence / trust factor indicating how credible the associated lock is. It should be appreciated that each of the remaining lock lifetime information, current brightness information, and confidence / trust factor may be interchangeably stored in any of the red, green, and blue channels of lock state 620 .

[0101] In one or more specific implementations, the locking component 520 determines whether to create a lock for a pixel by comparing a neighborhood (e.g., 3×3 pixels) of luminance values ​​associated with a pixel of the current frame 502 to a luminance difference threshold. In other words, the locking component 520 determines whether the relative luminance difference between the pixel and the neighborhood is below the luminance difference threshold. The luminance difference can be characterized as max(A,B) / min(A,B), where A and B are luminance / luma values. An example of a luminance difference threshold is a given percentage of similarity to the central luminance of a neighborhood (e.g., 3×3) of the pixel. The locking component 520 obtains the luminance value from the adjusted color buffer 608. The use of the neighborhood allows the locking component 520 to detect fine features (e.g., wires or chain-link fences) in the current frame 502 that should be locked in order to preserve details in the final super-resolution frame / image. If the color change difference (e.g., luminance difference) between the pixel and the neighborhood of luminance values ​​meets the luminance change threshold, the locking component 520 locks the pixel by creating an entry in the output lock state 620-2. For example, if the brightness change is lower than (or equal to) the brightness change threshold, the locking component 520 locks the pixel. Otherwise, the locking component 520 maintains the current lock state (e.g., unlocked state) of the pixel, which is also reflected in the output lock state 620-2 in at least some specific implementations. In more detail, the locking component 520 compares the brightness value of the center pixel in the 3×3 area with the brightness values ​​of the neighboring pixels in the 3×3 (or other) neighborhood. If the relative brightness difference between the center pixel and the neighboring pixels is lower than the brightness difference threshold, the locking component 520 sets a bit in the mask. If the brightness of the center pixel is outside the brightness range of all neighboring pixels, the locking component 520 considers the center pixel as a candidate for locking. After the locking component 520 has compared the brightness of the center pixel with all neighboring pixels, the locking component 520 determines whether the resulting mask is noisy, that is, whether it does not have a 2×2 (or other) area with all bits set. If the mask is noisy and the brightness of the center pixel is outside the brightness range of all neighboring pixels, the lock component 520 generates a lock for the center pixel and stores the lock in the output lock state 620-2.

[0102] In addition to creating a new lock, the lock component 520 (or another component of the amplifier 332) also updates the lock generated for the previously enlarged frame in the input lock state 620-1. For example, the lock component 520 decrements the red channel value of the pixel lock generated for the previously enlarged frame in the input lock state 620-1 by the initial pixel lock length divided by the total length of the jitter sequence (of the color buffer / camera). For example, if the initial length of the lock is initially set to 1, then subtracting 1 / jitterSequenceLength from the remaining life of the lock causes the remaining life of the lock to become 0 after one iteration of the jitter sequence. In another specific implementation, the lock component 520 (or another component of the amplifier 332) dynamically decrements / degrades the red channel value of the pixel lock in the input lock state 620-1 based on the contribution of the locked pixel to the final color for the current frame. When the lock reaches zero (or another threshold), the lock component 520 considers the lock expired and releases the lock. In one or more implementations, the initial length of the lock is modified by a trust / confidence factor that indicates how confident the lock component 520 is that the associated pixel should be locked, which affects how long the pixel will be locked. The lock component 520 also updates one or more locks in the input lock state 620-1 by releasing the lock, as described below.

[0103] In the reprojection and accumulation stage 611, the reprojection and accumulation component 522 takes as input the deocclusion mask 618, the dilated motion vector buffer 614, the reactive mask 602, the output buffer 506-1 of the previous frame, the current luma texture 606, the luma history 610, the adjusted color buffer 608, and the lock state 620. The reprojection and accumulation component 522 processes this input to generate an output buffer (texture) 506-2 for the current frame 502 at the target rendering resolution / size, and also generates a reprojected pixel lock (texture) 622 from the previous frame that can be mapped to the current frame 502. As described below, the reprojection and accumulation component 522 accumulates the reprojected color data from the previous frame together with the upsampled color data from the current frame 502, and stores the accumulated color data in the output buffer 506-2. In one or more implementations, the output buffer 506-2 is a texture used internally by the upscaling stage 332 and is different from the rendering buffer 626 generated by the sharpening stage 613. In addition, in one or more implementations, the output buffer 506-2 is a portion of the output buffer 506, which is represented as an array of two additional textures (e.g., output buffer 506-1 and output buffer 506-2) consumed by the reprojection and accumulation stage 611. In odd frames, the output buffer 506-1 is read and the output buffer 506-2 is written (or vice versa). In even frames, the output buffer 506-2 is read and the output buffer 506-1 is written (or vice versa).

[0104] The reprojection and accumulation phase 611 includes multiple sub-phases, such as Fig. 9 As shown. In one or more specific implementations, these substages include a shading change detection substage 901, an upsampling substage 903, a reprojection substage 905, a lock update substage 907, a color correction substage 909, one or more tone mapping substages 911 (illustrated as tone mapping substage 911-1 and tone mapping substage 911-2), an accumulation substage 913, and an inverse tone mapping substage 915. In at least some specific implementations, the reprojection and accumulation component 522 includes one or more subcomponents 902 to 914 that perform operations associated with one or more of the reprojection and accumulation substages 901 to 913. In one or more specific implementations, the reprojection and accumulation stage 611 is performed on a per-frame basis and is performed once per pixel in parallel for each output / rendering resolution pixel. Fig. 9 , and executes in parallel for each pixel of the enlarged frame 508 Fig. 9 The data flow is shown in the example. Fig. 9, a "2×2 bilinear" reference indicates that bilinear sampling is performed at the corresponding stage, a "5×5 Lanczos" reference indicates that Lanczos resampling is performed at the corresponding stage, and a "1" reference indicates that point sampling is performed. However, it should be understood that other types of sampling may also be applicable.

[0105] As described below, the reprojection and accumulation component 522 performs a number of operations via one or more of the subcomponents 902-914, including: upsampling the color buffer 320-1 for the current frame 502; reprojecting the (historical) color data 922 and pixel locks provided by the output buffer 506-1 of the previous frame as if they were viewed from the perspective of the current camera; cleaning the reprojected color data 922; accumulating final historical color data / values ​​928 ( Fig. 9 ) and the upsampled color data 918 for the current frame 502; and (optionally) inverse tone mapping the super-resolution color values ​​930 generated by the reprojection and accumulation stage 611 for sharpening.

[0106] At the beginning of the reprojection and accumulation phase 611, the shading change detection sub-phase 901 obtains the brightness history 610, the current brightness texture 606, and the input lock state 620-1. The shading change detection component 902 processes this input to evaluate each pixel of the current frame 502 for detecting changes in the shading of the pixel. In one or more specific implementations, the shading change detection component 902 also performs texture filtering, such as bilinear filtering, on these inputs. In one example, the shading change detection component 902 uses the lock state 620-1 of the current pixel in the previously enlarged frame to determine whether the pixel is locked or unlocked. If the pixel is locked, the shading change detection component 902 determines whether the shading of the pixel has changed by comparing the brightness (Y) value of the pixel when the lock was created with a shading change threshold. In one or more specific implementations, the shading change detection component 902 compares the brightness of the pixel with the average brightness of a neighborhood (e.g., 32×32) of the pixel in the current frame 502. If the difference between the brightness of the pixel and the average brightness of the neighborhood of the pixel satisfies (e.g., is greater than or alternatively equal to) the shading change threshold, the shading change detection component 902 determines that the shading of the pixel has changed. In one or more specific implementations, the shading change detection component 902 obtains the brightness of the pixel from the green channel of the input lock state 620-1 for the pixel. If the pixel is unlocked, in one or more specific implementations, the shading change detection component 902 determines whether the shading of the pixel has changed based on the brightness value of the pixel in the current frame and the historical brightness value of the neighborhood of the pixel or the historical pixel. For example, if the difference between the brightness value of the pixel in the current frame and the historical brightness value of the pixel (or the neighborhood of the historical pixel) satisfies the shading change threshold, the shading change detection component 902 determines that the shading of the pixel has changed. In one or more specific implementations, the shading change detection component 902 obtains the current brightness value for the pixel from the current brightness texture 606 and obtains the historical brightness value of the pixel from the brightness history 610. The shading change detection component 902 generates shading change data 924, which includes, for example, a bit / flag indicating whether the shading of the pixel has changed. Shading change data 924 is received as input by upsampling sub-stage 903 , lock update sub-stage 907 , and color correction sub-stage 909 .

[0107] The upsampling component 904 of the upsampling substage 903 takes as input the shading change data 924 from the shading change detection component 902 and the adjusted color buffer 608 for upsampling the adjusted color buffer 608. In one or more implementations, the upsampling component 904 uses the shading change data to change the shape of the filter kernel, which potentially causes lower or higher sample weights to be used in the accumulation substage 913. Upsampling the adjusted color buffer 608 involves interpolating between existing pixels of the adjusted color buffer 608 to obtain estimates of the values ​​of these existing pixels at new pixel locations. For example, if the current frame 502 is being upsampled from 1920×1080 pixels to 3840×2160 pixels, the upsampling component 904 interpolates between the original 1920×1080 pixels to estimate the color values ​​for the new pixels that are upsampled to the higher 3840×2160 pixel resolution. In at least some implementations, the upsampling component 904 implements Lanczos resampling to upscale the pixels of the adjusted color buffer 608, but one or more different scaling techniques or algorithms (such as Sinc resampling, nearest neighbor interpolation, bilinear algorithm, bicubic algorithm, box sampling, etc. or combinations thereof) may also be applicable.

[0108] In general, Lanczos resampling is an interpolation method used to calculate new values ​​for digitally sampled data. When used to resize a digital image, the Lanczos function indicates which pixels in the rendered (original) image and in what part make up each pixel in the upsampled image. For example, Fig.10 A plurality of rendered pixels 1002 from the current frame 502 is shown. In each rendered pixel 1002, a point P represents a set of low-resolution samples 1004 from which they will be resampled and for which Lanczos weights will be calculated. Each point S in the pixel 1002 represents a rendering resolution target pixel 1006 (also referred to herein as an upsampled pixel 1006) to be calculated based on resampling from the set of low-resolution samples 1004. The location of the upsampled pixel 1002-1 is used as the center of a Lanczos resampling kernel 1008 (such as a 5×5 Lanczos resampling kernel), which is resampled at Fig.101 is represented by a dotted grid. Lanczos functions 1010 (illustrated as Lanczos function 1010-1 and Lanczos function 1010-2) are located around the center of the upsampled pixel 1006-1 being calculated. Therefore, the color value for the upsampled pixel 1006-1 is determined by applying a Lanczos resampling kernel 1008 (e.g., Lanczos (x, 2)) to the low-resolution samples 1004-1 in the 5×5 neighborhood (grid) of pixels surrounding the upsampled pixel 1006-1. In other words, the weight of each low-resolution sample 1004 in the 5×5 neighborhood of the pixel is determined using the Lanczos resampling and the distance of the low-resolution sample 1004 to the rendering resolution target pixel 1006-1. The final color value of the upsampled pixel 1006-1 is based on the sum of the weights of each low-resolution sample 1004 in the 5×5 neighborhood of the pixel. In some instances, Lanczos resampling may introduce ringing artifacts. Therefore, in one or more implementations, the final color value of the upsampled pixel 1006-1 is clamped using, for example, a central 2x2 kernel range to mitigate any ringing artifacts.

[0109] Although the examples described above implement a 5×5 neighborhood of pixels during the Lanczos resampling process, in some instances, a 4×4 neighborhood is sampled due to the zero weighted contribution of pixels on the periphery of the 5×5 neighborhood. In addition, in one or more aspects, the specific implementation of the Lanczos kernel varies based on the GPU implemented. For example, for one or more GPU implementations, a lookup table (LUT) can be used to encode the sinc(x) function for the Lanczos kernel. Using the LUT provides a balance between arithmetic logic (ALU) operations and memory usage in the reprojection and accumulation stage 611. For example, for any given jitter, the Lanczos value for the sample used is the same for each pixel of the frame. Therefore, in one or more implementations, these Lanczos values ​​are pre-calculated and stored in the LUT. The LUT is passed as a texture to the shader that implements the upsampling component 904. Then, the pre-calculated Lanczos values ​​that may reside in the cache are repeatedly read by the upsampling component 904 in the shader instead of repeatedly calculating the Lanczos values. In one or more specific implementations, the LUT is read by a fixed function sampling block to reduce the ALU usage of the shader core. If the shader stage is not bandwidth-limited, implementing the LUT is faster than spending ALU cycles.

[0110] In at least some implementations, the upsampling component 904 also computes a YCoCg bounding box 1102 ( Fig.11 ).For example, Fig.11shows the current pixel (eg, Fig.10 1002-1) in the current frame 502. It should be understood that the bounding box 1102 also has a third dimension for Cg (not shown for illustration purposes). The bounding box 1102 is used during the color correction substage 909 described below. In one or more specific implementations, the upsampling component 904 generates a bounding box 1102 for each pixel in the current frame 502 by calculating the minimum and maximum values ​​of each channel for all pixels in the neighborhood (e.g., 3×3) surrounding the current pixel. The different coloring / patterns of the squares 1106 in the 3×3 neighborhood 1104 of the pixel represent the Y value of the pixel. The location of each square in the bounding box 1102 represents where the sample is located within the bounding box 1102. In at least some embodiments, the output of the upsampling sub-stage 903 includes upsampled (pixel) color data / values ​​918 for upsampled pixels (i.e., pixels upsampled to the rendering resolution) generated for the current frame 502, and includes a YCoCg bounding box 1102 generated for each upsampled pixel.

[0111] The reprojection component 906 of the reprojection substage 905 takes as input the dilated motion vector buffer 614, the input lock state 620-1, and the output buffer 506-1 of the previously upscaled frame (i.e., the frame previously processed by the upscaler 332). The reprojection component 906 processes this input to reproject the pixel lock information from the output color buffer from the output buffer 506-1 and the input lock state 620-1 from the previous frame, for pixels that do not have a new lock generated for the current frame 502, as viewed from the camera perspective of the current frame 502. For example, Fig.12 Frame 1202 is shown representing a previously upscaled frame. The reprojection component 906 samples the dilated motion vectors in the dilated motion vector buffer 614. The reprojection component 906 then applies the sampled motion vectors 1204 (illustrated as motion vector 1204-1 and motion vector 1204-2) to the output buffer 506-1 (e.g., color data) of the previously upscaled frame. Fig.12 The sampled two-dimensional motion vector 1204 is shown applied to the position of the current pixel 1206 of the output buffer 506-1 from the previously zoomed-in frame. In other words, the reprojection component 906 subtracts the sampled motion vector 1204 from the position coordinates of the current pixel to obtain the historical coordinates. The reprojection component 906 then samples the input lock state 620-1 and the historical color information (from the output buffer 506-1) using the historical coordinates (the translated pixel position).

[0112] In one or more specific implementations, the reprojection component 906 performs Lanczos resampling to sample the neighborhood of pixels surrounding the translated pixel location. Fig.12 A 5×5 grid 1210 of pixels surrounding the translated pixel location 1206-1 is shown with a Lanczos (x, 2) resampling kernel 1208 applied. In one or more specific implementations, the reprojection component 906 reprojects the lock using bilinear interpolation at the translated pixel location. Thus, the output of the reprojection substage 905 is a presentation resolution image that includes all data from the previous frame that can be mapped to the current frame 502. In other words, the output of the reprojection substage 905 is the reprojected color data 922 of the pixels in the previously upscaled frame that can be mapped to the current frame 502 and the reprojected pixel lock 622. For example, the remaining lifetime from the red channel, the current brightness of the current frame 502 from the green channel, and the confidence / trust factor from the blue channel (if implemented) of the input lock state 620-1 for pixels that do not have a new lock generated for the current frame 502 are reprojected / written as the reprojected lock 622. In one or more implementations, the reprojected lock 622 refers to the reprojected lock information written to the output lock state 620-2 of the pixel or the reprojected lock information stored in the intermediate lock texture. In at least some implementations, the reprojection component 906 determines whether a new lock has been generated for the pixel based on the output lock state 620-2 of the pixel including a negative remaining life value. In one or more implementations, if the input lock state 620-1 of the pixel indicates that the pixel has only been locked once (e.g., the remaining life value is "1"), the lock is not valid, and the reprojection component 906 does not perform a reprojection operation for the pixel. In at least some implementations, the reprojected pixel lock 622 is reprojected into the output lock state 620-2 and used as a lock for pixels that are not updated with new sampling information in the current frame 502, while the pixel lock generated during the lock stage 609 is used for pixels with updated samples in their area. It should be understood that the lock reprojection process of the reprojection substage 905 is not limited to image magnification. For example, the lock reprojection process is also applicable to TAA and other technologies.

[0113] The lock update component 908 of the lock update sub-stage 907 takes as input the de-occlusion mask 618, the re-projected lock 622, and the shading change data 924. The lock update component 908 processes this input to generate an updated set of pixel locks 926. Here, updating the pixel locks includes identifying a set of pixel locks from the re-projected lock 622 to be passed to the color correction sub-stage 909. For example, if the lock update component 908 determines that the de-occlusion mask 618 indicates that the pixel associated with the re-projected pixel lock has been occluded, the lock update component 908 releases the lock. In another example, the lock update component 908 determines that the shading change data 924 indicates that the shading of the pixel associated with the re-projected pixel lock has changed by more than a shading change threshold, and the lock update component 908 releases the lock. In one or more specific implementations, the lock update component 908 also updates the re-projected lock to reflect the attenuation and the released lock.

[0114] The lock update component 908 then determines which of the remaining re-projected locks are credible for the current frame 502. In one or more specific implementations, the lock update component 908 determines the credibility of the pixel lock by comparing the brightness value associated with the pixel lock (i.e., the brightness value of the pixel when the pixel lock was created) with the brightness values ​​in the current brightness texture 606, which represent the average brightness of the scene in the neighborhood of pixels around the pixel associated with the current pixel lock. If the brightness separation between the compared brightness values ​​exceeds (or is equal to) the brightness separation threshold, the lock update component 908 determines that the pixel lock is not credible and releases the lock. However, if the brightness separation between the compared brightness values ​​is less than (or equal to) the brightness separation threshold, the lock update component 908 determines that the pixel lock is credible and maintains the pixel lock.

[0115] In one or more implementations, the lock update component 908 also processes the input to identify a new pixel lock that was created during the lock phase 609 of the current frame 502 and was not reprojected. As described above, in at least some implementations, the lock update component 908 identifies the new pixel lock based on the red channel of the output lock state 620-2 having a negative value or based on the remaining lock lifetime being equal to the initial lock lifetime. When the new pixel lock is identified, the lock update component 908 does not perform the lock release operation described above for the new pixel lock. However, in one or more implementations, the lock update component 908 uses the current brightness information from the current brightness texture 606 to update the green channel of the output lock state 620-2 for the valid new pixel lock (e.g., at least as a pixel lock for the second pixel lock of the pixel). In one or more implementations, the lock update component 908 also updates the confidence / trust factor of the pixel lock stored in the blue channel of the lock state 620 of the new valid pixel lock and the reprojected pixel lock (if implemented). For example, in at least some implementations, the lock update component 908 increases the confidence / trust factor of the new pixel lock and the reprojected lock based on their current upsampling weight contributions. The increase value is in the normalized range [0,1] and is determined by dividing the current frame upsampling weights (e.g., the Lanczos weights calculated in the upsampling substage 903) by a constant value (e.g., the maximum cumulative weight). The increase value is added to the current confidence / trust factor stored in the blue channel of the lock state 620. In at least some implementations, the confidence / trust factor is clamped to the [0,1] range.

[0116] In one or more specific implementations, the lock update component 908 also reduces the lock lifetime of the new pixel lock and the reprojected lock. For example, the lifetime of the reprojected lock or new pixel lock (in the 2× magnification example) is reduced by a factor of approximately 1 / 32. In another example, the lifetime of the reprojected lock or new pixel lock is dynamically reduced based on the contribution of the locked pixel to the final color for the current frame (i.e., the current upsampling weight contribution). In at least some specific implementations, the lock update component 908 calculates the reduction value by dividing the current frame upsampling weight by a factor based on the jitter sequence length and the average upsampling kernel weight. In one or more specific implementations, the reduction value is within the normalized range [0,1]. Then, the lock update component 908 updates the remaining lifetime of the pixel lock stored in the red channel of the lock state 620 by subtracting the reduction value from the current remaining lifetime of the pixel lock. In one or more embodiments, the resulting value is clamped to the [0,1] range to avoid negative lifetime values.

[0117] The output of the lock update substage 907 is an updated lock set 926 (i.e., lock state 620), which includes the reprojected pixel locks 622 that have been determined to be credible, or alternatively includes a set 622 of reprojected locks with an indicator associated with each lock identifying whether the lock is credible or untrusted. In one or more specific implementations, the updated pixel lock set 926 also includes pixel locks that have been newly created and are valid for the current frame 502. In at least some specific implementations, each reprojected pixel lock in the updated lock set 926 includes an updated remaining lifetime, brightness value, and confidence / trust factor, and each newly generated valid lock includes an updated brightness value and confidence / trust factor as described above. The updated output lock state 620-2 of the new pixel lock is then stored in the output buffer 506-2 for use during upsampling of the next frame. It should be understood that the lock update substage 907 is not limited to image magnification. For example, the lock update substage 907 can also be applied to TAA and other techniques.

[0118] The updated lock set 926 is then passed to the color correction substage 909 as input. The color corrector component 910 of the color correction substage 909 takes as input, for example, the upsampled color data 918 for the upsampled pixels of the current frame 502, the color data 922 for the reprojected pixels of the previously upscaled frame, the updated lock set 926, the deocclusion mask 618, the brightness stability factor (e.g., the .w channel of the brightness history 610), the shading change data 924, and the lock contribution factors of the new or reprojected locked pixels. The lock contribution factors are calculated based on the weights of the upsampled color data 918, the optional transparency and compositing masks, and the optional reactive mask 602. The color corrector component 910 processes the inputs to determine the final historical color data / value 928 for each upsampled pixel of the current frame 502, and the contribution of the final historical color data / value to the final upscaled pixel color. In one or more specific implementations, when determining the final historical color value 928 for each upsampled pixel, the color corrector component 910 reduces the influence of the historical samples based on various factors, including, among others, the lock state or deocclusion. In other words, in some instances, the color corrector component 910 modifies the historical color values ​​to align with the shading of the current frame 502. For example, if a pixel position has been (partially) deoccluded, this means that the pixel position was occluded in the previously upscaled frame. Therefore, the historical color data for the pixel position is no longer valid for improving the quality of the current pixel. Therefore, the color corrector component 910 reduces the influence of the historical samples of the pixels that have become deoccluded to avoid artifacts, such as ghosting. In this example, the color corrector component 910 reduces the influence of the historical samples by modulating the color values ​​of the historical samples by the deocclusion mask 618.

[0119] Fig.13 Another example is shown, in which the corrector component 910 is based on historical samples (S h ) 1302 from the color bounding box 1102 of the upsampled pixel to reduce the influence of these historical samples. In this example, the greater the distance (d), the more the influence of the historical samples 1302 is reduced by the color corrector component 910. In one or more specific implementations, if the color corrector component 910 determines that the historical samples are locked based on the updated lock set 926 and also determines that the reactive mask 602 or the de-occlusion mask 618 does not invalidate the lock, the color corrector component 910 increases the influence of the historical color data of the historical samples relative to the upsampled color data 918.

[0120] In one or more specific implementations, at least one of the upsampled color data 918 or the final historical color data 928 is passed to a tone mapping substage 911 (illustrated as a tone mapping substage 911-1 and a tone mapping substage 911-2). For example, if the upsampled color data 918 or the final historical color data 928 includes high dynamic range data, the upsampled color data 918 or the final historical color data 928 is passed to the tone mapping substage 911, respectively. Then, a tone mapping component 912 (illustrated as a tone mapping component 912-1 and a tone mapping component 912-2) applies one or more local tone mapping operations to the upsampled color data 918 or the final historical color data 928 to reduce artifacts (such as firefly artifacts). In one or more specific implementations, tone mapping ensures that color values ​​are in the range from 0 to 1 by dividing the color by the maximum channel value, for example, Color.rgb=color.rgb / (1+max(max(color.r,color.g),color.b), where max is the maximum channel value, color.r is the red channel value, color.g is the green channel value, and color.b is the blue channel value.

[0121] The accumulation component 914 of the accumulation substage 913 takes as input the upsampled color data 918 and the final historical color data 928 for each pixel of the current frame 502, which may or may not be tone mapped. For each upsampled pixel of the current frame 502, the accumulation component 914 accumulates (e.g., blends) the upsampled color data 918 of the upsampled pixel with the final historical color data 928 to jointly form a final accumulated super-resolution color value 930 of the upsampled version of the current frame 502. An implicit result of the temporal accumulation of multiple frames during the accumulation substage 913 is that the resulting final accumulated super-resolution color value 930 is anti-aliased. In one or more specific implementations, the accumulation component 914 blends the upsampled color data 918 of the current frame 502 with the final historical color data 928 at a relatively low linear interpolation factor so that a small amount of color data from the current frame 502 is included in the super-resolution color value 930 of the enlarged frame 508. In other words, the final historical color data 928 determined for each upsampled pixel contributes more to the final accumulated super-resolution color value 930 than the upsampled color data 918 of the current frame 502. Mixing the two color values ​​can be characterized as follows: Mixed value = A*(1-W) + B*W, where A is the final historical color data 928, B is the color data from the current frame 502, and W is the contribution weight. In one or more specific implementations, the contribution weight of the upsampled color data 918 can be increased based on the reactive mask 602 that the accumulation component 914 also takes as input. The reactive mask 602 indicates to the accumulation component 914 where the accumulation component should reduce the accumulation component's reliance on historical information when synthesizing the current upsampled pixel, and instead allow the color data 918 of the upsampled pixel to contribute more to the final accumulated super-resolution color value 930 (also referred to herein as "super-resolution color value 930" for simplicity). For example, for a given pixel of the current frame 502, the accumulation component 914 multiplies the reactive value associated with the pixel from the reactive mask 602 by the contribution weight W to increase the contribution weight of the upsampled color data 918. Therefore, mixing two color values ​​considering the reactive mask 602 can be characterized as follows: Mixed Value = A*(1-(R*W)+B*(R*W), where A is the final historical color data 928, B is the color data from the current frame, and R is the reactive value from the reactive mask 602.

[0122] The super-resolution color values ​​930 are stored in the output buffer 506-2, representing an upscaled version of the current frame 502 at the rendering resolution. In other words, the output buffer 506-2 includes the upscaled frame 508. In one or more specific implementations, the output buffer 506-2 is used as the output buffer 506-1 as input by the amplifier 332 when processing the next frame. In one or more specific implementations, the super-resolution color values ​​930 are passed to the inverse tone mapping substage 915 before being stored in the output buffer 506-2. In these specific implementations, the inverse tone mapping component 916 performs inverse tone mapping on the super-resolution color values ​​930. The inverse tone mapping process reverses the tone mapping process described above so that the color range of the super-resolution color values ​​930 matches the color range of the input frame 502. In one or more specific implementations, the inverse tone mapping process can be characterized by Color.rgb=color.rgb / (1.f−max(max(color.r, color.g), color.b)), where f indicates a floating point number, max is the maximum channel value, color.r is the red channel value, color.g is the green channel value, and color.b is the blue channel value.

[0123] refer to Figure 6 , the sharpening component 524 of the sharpening stage 613 takes as input the output buffer 506-2 including the enlarged frame 508. The sharpening component 524 performs one or more sharpening operations on the pixels of the enlarged frame 508 represented by the output buffer 506-2. The sharpening component 524 stores the sharpened enlarged frame 508 in the rendering buffer 626. In one or more specific implementations, the sharpening component 524 performs robust contrast adaptive sharpening (RCAS). However, other image sharpening techniques may also be applicable. RCAS generates additional clarity and sharpness in the final enlarged image, and when converting local contrast into a variable amount of sharpness, solves for the maximum local sharpness possible before cropping. RCAS also has a built-in process to limit the sharpening of potential noise in the image. In more detail, RCAS uses a 5-tap filter 1404 ( Fig.14 ) for the data 1402 sampled from the output buffer 506-2 ( Fig.14 ) to perform operations, such as Fig.14 During the RCAS process, the sharpening component 524 calculates the weight w based on the following equation:

[0124]

[0125] Where S m is the middle sample, S n YesS m The northern sample, S e YesSm The eastern sample, S s YesS m The sample from the south, and S w YesS m West side sample.

[0126] The sharpening component 524 selects w0 or w1 based on which weight w does not cause clipping, and further constrains the weight w. The sharpening component 524 also multiplies by a "sharpening amount", which is a sharpness value provided by the application 206 that changes the perceived sharpness of the output image. The RCAS process performs a high pass filtering process that is normalized for local contrast and then shaped. This process is used as a noise detection filter to reduce the effect of RCAS on grain and focus on real edges. The RCAS process also supports pass-through alpha. After the sharpening process has been completed, the sharpening component 524 stores the data for the upscaled frame 508 in the rendering buffer 626. In at least some specific implementations, one or more post-amplification post-processing stages 311-2 perform post-processing operations on the upscaled frame 508 stored in the rendering buffer 626. The post-processed upscaled frame 508 is then output by the graphics processing pipeline 214 for presentation on the display device 118. Alternatively, the upscaled frame 508 is output by the graphics processing pipeline 214 for presentation on the display device 118 after sharpening without performing any post-amplification post-processing operations thereon.

[0127] Fig.15 An overview of an example method 1500 for performing spatial upscaling of a rendered frame 502 by reconstructing a high-resolution image representing the rendered frame 502 of a video stream using temporal feedback is illustrated in flowchart form. It should be understood that the process described below with respect to method 1600 has been previously described above with reference to Figures 6 to 14Described in more detail. At box 1502, the amplifier 332 obtains a first frame of a video stream rendered at a first resolution. The first frame is defined by a first plurality of pixels associated with a first set of color data. At box 1504, the amplifier obtains a second frame of the video stream that is amplified to a second resolution greater than the first resolution. The second frame is defined by a second plurality of pixels associated with a second set of color data. At box 1506, the amplifier locks one or more pixels in the first plurality of pixels. At box 1508, the amplifier 332 upsamples the first plurality of pixels to a second resolution. The upsampling generates upsampled color data for the upsampled first plurality of pixels based on the first set of color data. At box 1510, the amplifier 332 accumulates the upsampled color data with the second set of color data to generate final color data for the upsampled first plurality of pixels. The upsampled color data associated with one or more locked pixels is maintained during the accumulation of the upsampled color data of at least one of the rendered frame or subsequent frames. In addition, the color data associated with the pixel lock in the second set of color data contributes more to the final color data than the corresponding color data in the upsampled color data. At block 1512 , the amplifier 332 stores the upsampled first plurality of pixels along with the final color data as an upscaled frame representing the first frame at the second resolution.

[0128] Fig.16 and Fig.17 An overview of another example method 1600 for performing spatial upscaling of a rendered frame 502 by reconstructing a high-resolution image representing a rendered frame 502 of a video stream using temporal feedback is illustrated together in flowchart form. It should be understood that the process described below with respect to method 1600 has been previously described with reference to Figures 6 to 14 1 is described in more detail. At block 1602, the rendering stage 313 of the graphics pipeline 214 renders the frame 502 at a first resolution. At block 1604, the rendered frame 502 is passed to the super-resolution amplifier 332 of the graphics pipeline 214. At block 1606, the auto-exposure stage 601 of the amplifier 332 processes the color buffer 320-1 of the rendered frame 502 to generate an exposure texture 604 including an exposure value determined for the current frame and a current brightness texture 606 including a current brightness value for the rendered frame 502. At block 1608, the input color adjustment stage 603 of the amplifier 332 processes the exposure texture 604 and the color buffer 320-1 of the rendered frame 502 to generate an adjusted color buffer 608 and a brightness history texture 610, which includes historical brightness information / values ​​for the rendered frame 502.

[0129] At block 1610, the reconstruction and dilation stage 605 of the amplifier 332 processes the depth buffer 322-1 and the motion vector buffer 504 of the rendered frame 502 to generate a previous depth buffer 616, a dilated depth buffer 612, and a dilated motion vector buffer 614 for the rendered frame 502. At block 1612, the depth clipping stage 607 of the amplifier 332 processes the previous depth buffer 616, the dilated depth buffer 612, and the dilated motion vector buffer 614 to generate a deocclusion mask 618 for the rendered frame 502. At block 1614, the pixel locking stage 609 of the amplifier 332 processes the adjusted color buffer 608 to lock and generate a locked state texture 620 for one or more pixels of the rendered frame 502.

[0130] At block 1616, the reprojection and accumulation stage 611 of the upscaler 332 processes the deocclusion mask 618, the dilated motion vector buffer 614, the reactive mask 602 associated with the pixels of the rendered frame 502, the output buffer 506-1 for the previously upscaled frame, the current luminance texture 606, the luminance history texture 610, the adjusted color buffer 608, and the lock state texture 620 to generate a super-resolution (and anti-aliased) upscaled frame 508 representing the rendered frame. The upscaled frame 508 has a second resolution that is greater than the first resolution of the rendered frame 502. At block 1618, the sharpening stage 613 of the upscaler 332 sharpens the upscaled frame 508. At block 1620, the sharpened upscaled frame 508 is stored in the rendering buffer 626 for presentation on the display device 118. In one or more implementations, additional post-processing operations are performed on the upscaled frame 508 before presenting the upscaled frame 508 on the display device 118. Flow returns to block 1602 , and the next rendered frame is processed by the amplifier 332 .

[0131] Fig.18 and Fig.19 Together with the flowchart Fig.17 A more detailed method 1800 of the reprojection and accumulation process illustrated in block 1616 of FIG. It should be understood that the process described below with respect to method 1800 has been previously described with reference to FIG. Figures 9 to 141 is described in more detail. At block 1802, the shading change detection substage 901 processes the lock state texture 620, the current brightness texture 606, and the brightness history texture 610 to generate shading change data 924 for each pixel of the rendered frame 502. At block 1804, the upsampling substage 903 processes the adjusted color buffer 608 to upsample the pixels of the rendered frame 502 at a second resolution (i.e., the target rendering resolution). The upsampling substage 903 also generates a color bounding box 1102 for each upsampled pixel. At block 1806, the reprojection substage 905 processes the dilated motion vector buffer 614, the lock state texture 620, and the output buffer 506-1 of the previously upscaled frame to reproject pixel data (e.g., color data 922) and pixel lock 622 that can be mapped to the rendered frame 502 from the previously upscaled frame.

[0132] At block 1808, the lock update stage 907 processes the deocclusion mask 618, the reprojected pixel locks 622, and the shading change data 924 to generate an updated pixel lock set 926 including a trusted lock. Locks determined to be untrustworthy are released. At block 1810, the color correction substage 909 processes the upsampled pixel data (e.g., color data 918), the color bounding box 1102 for the upsampled pixels, the updated pixel lock set 926, the deocclusion mask 618, the reprojected pixel color data 922, and the shading change data 924 to determine the final historical color data / value 928 for each upsampled pixel.

[0133] At block 1812, one or more tone mapping substages 911 tone map the final historical color data 928 and the color data 918 of the upsampled pixel. However, in at least some implementations, the tone mapping operation is optional. At block 1814, the accumulation substage 913 processes the final historical color data 928 and the color data 918, which may have been tone mapped, to generate a final accumulated super-resolution color value 930 for the upsampled pixel. At block 1816, the inverse tone mapping substage 915 inverse tone maps the super-resolution color value 930. However, in at least some implementations, the inverse tone mapping operation is optional. At block 1818, the super-resolution color value 930, which may have been inverse tone mapped, is stored in an output buffer 506-2, which represents the enlarged frame 508 having a second resolution and corresponding to the rendered frame 502.

[0134] Fig. 20 The overall process for generating a pixel lock for one or more pixels is illustrated in flowchart form. It should be understood that the process described below with respect to method 2000 has been previously described with reference to Fig. 9Described in more detail. At box 2002, the locking stage 609 obtains a first frame of the video stream. The first frame is defined by a first plurality of pixels associated with a second set of color data. At box 2004, the locking stage 609 determines that a pixel in the first plurality of pixels includes high frequency information. At box 2006, in response to determining that the pixel includes high frequency information, the locking stage 609 generates a first pixel lock for the pixel so that the color data associated with the pixel is maintained during the color accumulation process for subsequent frames. At box 2008, the locking stage 609 obtains pixel lock information for a second frame of the video stream. The second frame is defined by a second plurality of pixels. The pixel lock information identifies at least a second pixel lock associated with at least one pixel in the second plurality of pixels. At box 2010, the locking stage 609 decrements the remaining life of the at least second pixel lock.

[0135] Fig.21 The flowchart is shown in Fig.17 A more detailed method 2100 of the pixel locking process illustrated in block 1614 of FIG. It should be understood that the process described below with respect to method 2100 has been previously described with reference to FIG. Fig. 9 6 is described in more detail. At block 2102, the lock stage 609 of the amplifier 332 takes as input the adjusted color buffer 608 and the lock state texture 621 for the previously enlarged frame. At block 2104, for each pixel of the rendered frame 502, the lock stage 609 compares the defined neighborhood of brightness values ​​obtained from the adjusted color buffer 608 to a brightness threshold. At block 2106, the lock stage 609 determines whether the brightness threshold has been met. At block 2108, if the brightness threshold has not been met (e.g., the brightness change between the neighborhood pixel and the current pixel exceeds the brightness threshold), the lock stage 609 stage does not create a lock for the current pixel. The process proceeds to block 2106. At block 2110, if the brightness threshold has not been met (e.g., the brightness change between the neighborhood pixel and the current pixel is below the brightness threshold), the lock stage 609 creates a lock for the current pixel. At block 2112, as part of creating the lock, the lock stage 609 stores the remaining life of the lock in the red channel of the lock state texture 620 for the current pixel. At the initial creation of a lock, the pixel has the full life of the lock. At block 2114, the lock stage 609 stores the brightness value of the pixel when the lock was created in the green channel of the lock state texture 620 for the current pixel. At block 2116, for each locked pixel of the previously zoomed-in frame, the lock stage 609 updates the remaining life of the lock in the red channel of the lock state texture 620 for that pixel.

[0136] Fig. 22 The flowchart is shown in Fig.18 The reprojection illustrated in block 1806 and Fig.18A more detailed method 2200 of the lock update process illustrated in block 1808 of FIG. It should be understood that the process described below with respect to method 2200 has been described above with reference to FIG. Fig. 9 and Fig.12 6-1 to generate reprojected color data 922 for pixels of the previously enlarged frame. At block 2204, the reprojection substage 905 samples the dilated motion vectors and applies the samples to the lock state texture 620 of the previously enlarged frame to generate reprojected locks 622 for pixels of the previously enlarged frame. As described above, for pixels in the rendered (current) frame whose lock state texture 620 indicates that a new pixel lock is generated in the rendered frame, the dilated motion vector is not applied to the lock state texture 620 of the previously enlarged frame. At block 2206, the reprojection substage 905 passes the reprojected color data 922 as input to the color correction substage 909. At box 2208, the reprojection sub-stage 905 passes the reprojected lock 622 as input to the lock update sub-stage 907.

[0137] At block 2210, for each reprojected pixel lock, the lock update substage 907 determines, based on the deocclusion mask 618, whether the associated pixel has become occluded. At block 2212, if the pixel has become occluded, the lock update substage 907 releases the reprojected lock and processes the next reprojected pixel lock. At block 2214, if the pixel has not become occluded, the lock update substage 907 holds the reprojected pixel lock. Alternatively, or in addition to block 2212, at block 2216, the lock update substage 907 determines whether the shading of the pixel has changed by more than a shading change threshold. At block 2218, if the shading of the pixel has changed by more than a shading change threshold, the lock update substage 907 releases the reprojected lock and processes the next reprojected pixel lock. At block 2220, if the shading of the pixel has not changed by more than a shading change threshold, the lock update substage 907 holds the reprojected pixel lock. At block 2222, the lock update sub-stage 907 determines that the lock is credible. At block 2224, the lock update sub-stage 907 also updates any valid pixel locks generated for the rendered frame by, for example, updating the brightness stored in the green channel and updating the trust / confidence factor stored in the blue channel. Once all re-projected pixel locks and valid pixel locks for the rendered frame have been processed, the lock update sub-stage 907 passes the updated pixel lock set 926 to the color correction sub-stage 909.

[0138] In some specific implementations, the apparatus and techniques described above are implemented in a system including one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips). Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are generally represented as one or more software programs. One or more software programs include codes that can be executed by a computer system to manipulate the computer system to operate the code representing the circuit of one or more IC devices so as to perform at least a part of the process for designing or adjusting the manufacturing system to manufacture the circuit. The code may include instructions, data, or a combination of instructions and data. Software instructions representing design tools or manufacturing tools are generally stored in a computer-readable storage medium accessible to a computing system. Similarly, codes representing one or more stages of the design or manufacture of an IC device can be stored in and accessed from the same computer-readable storage medium or different computer-readable storage media.

[0139] Computer-readable storage media may include any non-transitory storage media or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact disks (CDs), digital versatile disks (DVDs), Blu-ray disks), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or a flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)).

[0140] In some specific implementations, certain aspects of the above-mentioned technology can be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions that are stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that manipulate one or more processors to perform one or more aspects of the above-mentioned technology when executed by one or more processors. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or other one or more non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats interpreted or otherwise executed by one or more processors.

[0141] It should be noted that not all activities or elements described above in the general description are required, a portion of a particular activity or device may not be required, and one or more additional activities may be performed, or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. In addition, these concepts have been described with reference to specific implementations. However, it is understood by those of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0142] Benefits, other advantages, and solutions to problems have been described above with respect to specific implementations. However, benefits, advantages, solutions to problems, and any features that may cause any benefit, advantage, or solution to appear or become more significant should not be interpreted as key, essential, or basic features of any or all claims. In addition, the specific implementations disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in different but equivalent ways that are obvious to those skilled in the art who benefit from the teachings herein. Except as described in the following claims, it is not intended to limit the details of the construction or design shown herein. Therefore, it is apparent that the specific implementations disclosed above may be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the following claims.

Claims

1. A method performed at an accelerated processing device, comprising: obtaining a first frame of a video stream rendered at a first resolution, wherein the first frame is defined by a first plurality of pixels associated with a first set of color data; obtaining a second frame of the video stream upscaled to a second resolution greater than the first resolution, wherein the second frame is defined by a second plurality of pixels associated with a second set of color data; upsampling the first plurality of pixels to the second resolution, wherein the upsampling generates upsampled color data for an upsampled first plurality of pixels based on the first set of color data; accumulating the upsampled color data with the second set of color data to generate final color data for the upsampled first plurality of pixels, wherein color data associated with a pixel lock in the second set of color data contributes more to the final color data than corresponding color data in the upsampled color data; as well as The upsampled first plurality of pixels are stored along with the final color data as an upscaled frame representing the first frame at the second resolution.

2. The method according to claim 1, further comprising: determining an exposure value for the first frame based on the first set of color data; as well as A current brightness of the first frame is determined based on the first set of color data.

3. The method of claim 2, wherein determining the exposure value comprises performing downsampling of the first set of color data to store the exposure value at a 1x1 mipmap level, and wherein determining the current brightness comprises performing downsampling of the first set of color data to store the current brightness at a half-resolution mipmap level.

4. The method according to any one of claims 1 to 3, further comprising: The first set of color data is converted from a first color space to a second color space before locking one or more pixels and upsampling the first plurality of pixels.

5. The method according to any one of claims 1 to 4, further comprising: dilating depth values ​​in the depth buffer of the first frame to generate a dilated depth buffer; dilating the motion vectors in the motion vector buffer of the first frame to generate a dilated motion vector buffer; for each pixel of the first plurality of pixels, estimating a depth value of the pixel in the second frame based on the dilated depth buffer and the dilated motion vector buffer; as well as The estimated depth value is stored in the previous depth buffer.

6. The method according to claim 5, further comprising: For each pixel in the first plurality of pixels, a deocclusion mask is generated based on the dilated depth buffer, the dilated motion vector buffer, and the previous depth buffer, wherein the deocclusion mask indicates an amount by which the pixel is deoccluded in the second frame.

7. The method according to any one of claims 1 to 6, further comprising: for each pixel in the first plurality of pixels, determining an amount by which shading of the pixel changes between the first frame and the second frame; as well as For each pixel of the first plurality of pixels, coloring change data is generated based on the coloring change amount determined for the pixel.

8. The method of claim 7, wherein determining the amount of change in shading of the pixel comprises: In response to the pixel being a lock, comparing a brightness of the pixel when the lock was created to a shading change threshold; as well as In response to the pixel being unlocked, a current brightness of the pixel is compared with historical brightness information associated with the second frame.

9. The method of any one of claims 1 to 8, wherein upsampling the first plurality of pixels to the second resolution comprises: For each pixel in the first plurality of pixels, a Lanczos resampling kernel is applied to a neighborhood of pixels around a center of the pixel.

10. The method according to claim 9, further comprising: For each pixel in the first plurality of pixels, a color space bounding box is generated based on the neighborhood of pixels around a center of the pixel.

11. The method according to any one of claims 1 to 10, further comprising: The second set of color data is reprojected onto the first frame to generate reprojected color data.

12. The method of claim 11, wherein reprojecting the second set of color data comprises: sampling a motion vector from a motion vector buffer associated with the first frame; as well as The sampled motion vector is applied to the second set of color data of the second frame.

13. The method according to claim 11, further comprising: A set of pixel locks associated with the second plurality of pixels is reprojected to generate reprojected pixel locks.

14. The method according to claim 11, further comprising: For each pixel of the upsampled first plurality of pixels, a final historical color value is determined based at least on the re-projected color data of the second frame.

15. The method of claim 14, wherein determining the final historical color value comprises: determining one or more pixels of the second plurality of pixels to be de-occluded by applying a de-occlusion mask to the re-projected color data; reducing the effect of color data from the reprojected color data associated with the one or more pixels determined to be de-occluded on the final historical color value; as well as The influence of the color data from the reprojected color data is reduced based on a distance of the color data from a color space bounding box generated for the upsampled first plurality of pixels.

16. The method of claim 14, wherein accumulating the upsampled color data with the second set of color data comprises: For each pixel in the upsampled first plurality of pixels, the upsampled color data for the pixel is accumulated with the final historical color value.

17. An accelerated processing device comprising a graphics processing pipeline for executing the method according to any one of claims 1 to 16.

Images