Content upscaling systems and applications using adaptive scanning

Abstract

The approaches presented here involve the generation, transmission, and upsampling of content for presentation using devices with varying graphics capabilities. Different computing devices may include integrated GPUs or other hardware with limited capacity, which may be unable to support more powerful graphics upscaling algorithms. Consequently, they may resort to a process such as hardware-implemented bilinear filtering for upscaling, resulting in lower-quality image rendering. An adaptive filter can be used to reduce the number of texture accesses required per pixel, potentially providing improved perceived quality and increased device support.The number of input samples required for an output pixel position can be adapted to the capacity of the device performing the upsampling, whereby a reduced number of samples or "tap"s per pixel can significantly reduce the resource capacity required to perform upscaling and interpolation using a Catnumm-Rom filter or Lanczos filter implemented as a GPU shader, and enable higher quality upscaling on systems with limited capacity.

Claims

[1] Computer-implemented method comprising: Determining one or more performance values ​​for a computing device; Receiving an input image having a first resolution suitable for display by the computing device, having a second resolution higher than the first resolution, is to be provided; Determining, based in part on the one or more performance values, a number of texture operations to be performed for at least one pixel position of an output image having the second resolution; Determining one or more output pixel values ​​for the at least one pixel position of the output image using an upsampling algorithm and by performing the determined number of texture operations; and Generating the output image at the second resolution using the one or more determined output pixel values. [2] The computer-implemented method of claim 1, further comprising: Setting one or more sample weight values ​​of the upsampling algorithm to a zero value to reduce the number of texture operations performed for the at least one pixel position to the specified number. [3] The computer-implemented method of claim 2, wherein increasing a number of sample weights set to a zero value decreases the number of texture operations performed on the at least one pixel to enable the upsampling algorithm to be executed on the computing device while satisfying one or more performance criteria. [4] A computer-implemented method according to any preceding claim, wherein the one or more performance values ​​relate to the performance of a graphics processor of the computing device. [5] A computer-implemented method according to any one of the preceding claims, wherein the upsampling algorithm accesses a central sample position and eight surrounding sample positions per pixel position, and wherein the number of texture samples to be performed is 3, 6 or 9. [6] A computer-implemented method according to any one of the preceding claims, wherein the sample positions for which the texture operations are performed correspond to at least one row or column including the central sample position. [7] A computer-implemented method according to any one of the preceding claims, wherein the upsampling algorithm is a Catmull-Rom algorithm. [8] A computer-implemented method according to claim 7, wherein the sample positions for which the texture operations are performed are selected to satisfy one or more conditions of an optimized implementation of the Catmull-Rom algorithm. [9] Processor comprising: one or more circuits for: Determining one or more performance values ​​for a computing device; Receiving an input image having a first resolution suitable for display by the computing device, having a second resolution higher than the first resolution, is to be provided; Determine, based in part on the one or more performance values, a number of texture operations to be performed for the at least one pixel position of an output image having the second resolution; Determining one or more output pixel values ​​for the at least one pixel position of the output image using an upsampling algorithm and by performing the determined number of texture operations; and Generating the output image at the second resolution using the one or more determined output pixel values. [10] The processor of claim 9, wherein the one or more circuits further apply the number of texture operations by setting one or more sample weights of the upsampling algorithm to a zero value to cause a corresponding sample position for a respective pixel position to not be accessed. [11] The processor of claim 9 or 10, wherein the one or more performance values ​​relate to the performance of a graphics processor of the computing device. [12] Processor according to claim 9 to 11, wherein the upsampling algorithm accesses a central sample position and eight surrounding sample positions per pixel position, and wherein the number of texture samples to be performed is 3, 6 or 9. [13] A processor according to any one of claims 9 to 12, wherein the upsampling algorithm is a Catmull-Rom algorithm. [14] A processor according to any one of claims 9 to 13, wherein the processor is comprised in at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for carrying out digital twinning operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for performing operations using one or more language models; a system for performing generative AI operations; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for synthetic data generation; a collaborative content creation platform for 3D assets; or a system implemented at least in part using cloud computing resources. [15] System comprising: one or more processors to generate an output image from a lower resolution input image using an upsampling algorithm and a number of texture samples for at least one pixel of the output image determined in part based on one or more performance measures of the system. [16] The system of claim 15, wherein the one or more processors further apply the number of texture operations by setting one or more sample weights of the upsampling algorithm to a zero value to cause a corresponding sample position for a respective pixel position to not be accessed. [17] The system of claim 15 or 16, wherein the one or more performance values ​​relate to the performance of a graphics processor of the system. [18] The system of any one of claims 15 to 17, wherein the upsampling algorithm accesses a central sample position and eight surrounding sample positions per pixel position, and wherein the number of texture samples to be performed is 3, 6, or 9. [19] The system of any one of claims 15 to 18, wherein the upsampling algorithm is a Catmull-Rom algorithm. [20] A system according to any one of claims 15 to 19, wherein the system comprises at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for carrying out digital twinning operations; a system for performing light transport simulation; a system for rendering graphical output; a system for performing operations using one or more language models; a system for performing generative AI operations; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing hardware tests using simulation; a system for synthetic data generation; a collaborative content creation platform for 3D assets; or a system implemented at least in part using cloud computing resources.

Images