Single pass rendering and upscaling

Abstract

A graphics processing system configured to process an image comprising a plurality of rendered pixels, the system being configured to divide the image into a plurality of tiles, each tile comprising a subset of the rendered pixels of the image and corresponding to a plurality of display pixels, the system comprising: a tile on-chip memory; and a processor, wherein the processor is configured to perform a processing pass for at least one rendered pixel in the image, comprising: rendering the rendered pixel by: determining a set of samples, each sample having a respective position in the rendered pixel and the positions collectively having a sampling density in the rendered pixel; shading the rendered pixel at a first resolution to give a shading result for the rendered pixel; writing a shading value for each of the set of samples to the tile on-chip memory based on the shading result; and storing an up-scaled representation of the rendered pixel in the tile on-chip memory by determining, from the shading values for the set of samples, a display pixel shading value for each of a plurality of display pixels overlapping the rendered pixel at a target resolution greater than the first resolution.

Description

TECHNICAL FIELD

[0001] This application relates to rendering and upscaling of images using anti-aliasing, for example for game rendering on tile-based mobile graphics processing units (GPUs). BACKGROUND

[0002] As consumer electronics resolutions increase, the demand on mobile GPUs also increases. Mobile GPUs need to render large screen products at 4K or 8K image resolutions. 4K screens are becoming more popular and 8K screens are now coming to market. These resolution requirements are shown. Figure 1

[0003] Using mobile GPUs to support rendering at 4K and 8K is very challenging for several reasons. Firstly, the pixel shading stage is already a bottleneck for most 3D games on mobile phone screens at 1080p resolution. Secondly, large screens mean that more pixels need to be shaded: 4K is 4 times the resolution of 1080p and 8K is 16 times the resolution of 1080p. Even modern high-end desktop GPUs can struggle with this additional shading workload. Therefore, it can not be possible to perform pixel shading natively at 8K resolution. Upscaling can therefore have to be performed after a low resolution image has been rendered at low cost.

[0004] Because mobile GPUs typically cannot perform pixel shading directly at 8K native resolution due to the large shading workload involved in the pixel shader, it is necessary to render at a low resolution (LR) and then upscale to the target high resolution (HR). Figure 2 The process of addressing the pixel shading workload by rendering at a reduced resolution and then upscaling to the required target resolution is shown schematically. Upscaling essentially means stretching the LR image to obtain the HR image. An "enlargement" needs to be made on the LR image so that it can fill the larger HR screen for display.

[0005] However, a drawback of this process is that when the game performs rendering at a reduced resolution, rasterization occurs at low resolution. Upscaling also means that the original imaging signal is under-sampled in a larger spatial domain. This therefore introduces aliasing into the rendered result. This ends up causing highly jagged object edges in the image and, in the worst case, can compromise the overall visibility of small objects, resulting in a sparkling and flickering effect. For example, spatial noise and temporal flicker can be introduced.

[0006] ​According to the Nyquist-Shannon theorem, the sampling rate must be equal to or higher than twice the highest frequency of the signal, otherwise aliasing occurs.

[0007] Mobile devices require real-time rendering performance, with high frame rates and low latency for user interactions for games. At the same time, they require low power consumption to extend battery life, and low heat dissipation to remain comfortable when held for long periods by the user. It can be a trade-off to render high quality and complex pixel shading at high screen resolutions, while ensuring that power consumption and heat dissipation are kept at very low levels.

[0008] One known upscaling method is recent reconstruction, as described at https: / / en.wikipedia.org / wiki / Image_scaling. In the simplest form of scaling, the original pixels are copied to fit the required higher resolution. This is a simple and fast upscaling method. However, the image quality is generally very poor, and the resulting image can appear jagged and blocky.

[0009] Deep Learning Super Sampling (DLSS) uses deep neural networks to extract multi-dimensional features of a rendered scene to construct a high-quality final image at high resolution, as described at https: / / www.techspot.com / article / 1873-radeon-image-sharpening-vs-nvidia-dlss / . DLSS forces games to render at a lower resolution (typically 1440p), and then uses artificial intelligence (AI) algorithms to infer what it would look like if rendered at a higher resolution (typically 4K). This can reduce aliasing. The method uses a dedicated tensor core on Nvidia GPUs for the AI algorithms. Less pixel shading is required, so rendering is faster. However, this method is invasive, requiring AI training on a per-game basis before release, involving large training sets and retraining for each game update. It is also not suitable for other GPUs without dedicated tensor cores, and DLSS has also resulted in blurry image quality for some test scenarios.

[0010] As described at https: / / www.amd.com / en / technologies / radeon-software-image-sharpening, Radeon Image Sharpening is a smart sharpening technology that can improve the visual quality of games without significantly degrading performance. The effect examines the high-contrast portions of any given scene in a game and manually renders more detail. Textures are sharpened globally, which can make a 1080p image appear closer to 1440p and displayed on an HR monitor after upscaling. Unlike DLSS, which must be implemented on a game-by-game basis, Radeon Image Sharpening is a post-processing of the image after upscaling, involving some spatial domain image filtering. It can be turned on or off for any game, does not need to be implemented on a game-by-game basis, and is non-intrusive. In most cases, a dedicated GPU with dedicated video memory and good image quality is less expensive. However, this method is unaware of which elements should be sharpened and which should not, and existing artifacts may be sharpened and become more visible.

[0011] As described at https: / / software.intel.com / en-us / articles / checkerboard-rendering-for-real-time-upscaling-on-intel-integrated-graphics, checkerboard rendering is a technique that produces full-resolution pixels with significantly reduced shading and minimal impact on visual quality. It develops a solution to address the problem of rendering content designed for higher target resolutions. It might take content with a target resolution of 1080p and render it at 540p (960x540), then use checkerboard (CBR) technology to upscale to 1080p by temporarily combining the results of two frames, each rendered with 2x multisample anti-aliasing (MSAA). This allows for 4x upscaling with good image quality at a low cost of 2x MSAA, and a reduced pixel shading rate (only a quarter of the target HR). However, the first column of pixels is not reconstructed, resulting in jagged edges. When rendering the current frame, the previous frame must maintain 2x MSAA. Therefore, this method has a bandwidth cost. If an object moves too quickly between two frames, it can cause temporal artifacts such as blurring and ghosting. It's also intrusive because the user must use a specific extension.

[0012] As described in https: / / microsoft.github.io / DirectX-Specs / d3d / VariableRateShading.html, Variable Rate Shading, or coarse pixel shading, is a mechanism that allows the allocation of rendering performance and power at different rates across a single frame of rendered image. Visually, in some cases, the shading rate can be reduced with little or no reduction in perceptible output quality, resulting in "at will" performance. This model extends supersampling with MSAA to the opposite "coarse pixel" direction by adding the novel concept of coarse shading. This is where shading can be performed at a coarser frequency than a pixel. That is, a group of pixels can be shaded as a single unit, and the result is then propagated to all samples in that group. The Coarse Shading Application Programming Interface (API) allows applications to specify the number of pixels belonging to a shading group. The coarse pixel size can be changed after the render target is assigned. Therefore, different parts of the screen or different draw passes can have different subsampling rates. Variable Rate Shading is a sophisticated solution that decouples shading rate from visibility. It allows for flexible changes to the shading rate with fine granularity. Different parts of the screen or different draw passes can have different subsampling rates. However, Variable Rate Shading (VRS) has more complex GPU thread scheduling logic, depending on the density map accessed in different parts of the screen. Due to the irregularity of the different shading rates between adjacent pixels, there is additional scheduling cost for so many concurrent GPU threads. Building and accessing the density map for each frame to specify and retrieve the shading rate map for each coarse pixel is costly.

[0013] As described in “MSAA-based coarse shading for power-efficient rendering on high pixel-density displays” by Mavridis, P. and Papaioannou, G., High Performance Graphics 2015, MSAA-based coarse shading employs a similar idea to reduce pixel shader calls by using MSAA samples, but it focuses on software implementation for existing GPUs and involves two rendering passes: the first pass renders to an intermediate render buffer with a low pixel count but a suitable number of MSAA samples, while the second pass reads the intermediate render buffer and performs sub-pixel MSAA sample-to-pixel mapping. The problem with this two-pass approach is that the intermediate render buffer storing the first pass’s output (which has a very large memory size due to the multiple MSAA samples per pixel) must be read from DDR memory into the GPU chip by the second pass, which is very bandwidth-intensive because the same intermediate render buffer is output and input during both passes.

[0014] In summary, the key technical issues with rendering and upscaling are that the algorithm may be too complex for mobile GPUs due to their limited computing power, memory bandwidth, battery capacity, and heat dissipation capabilities, and the image quality may be poor due to jagged edges in the final image after LR rendering and subsequent upscaling to HR.

[0015] A rendering and scaling method needs to be developed to overcome these problems. Summary of the Invention

[0016] According to a first aspect, a graphics processing system is provided, configured to process an image comprising a plurality of rendered pixels. The system is configured to divide the image into a plurality of tiles, each tile comprising a subset of the rendered pixels of the image and corresponding to a plurality of display pixels. The system includes: an on-chip tile memory; and a processor configured to perform a processing pass for at least one rendered pixel in the image, comprising: rendering the rendered pixel by: determining a set of samples, each sample having a respective position in the rendered pixel and these positions having a common sampling density in the rendered pixel; coloring the rendered pixel at a first resolution to give a coloring result of the rendered pixel; and writing a coloring value of each of the samples based on the coloring result into the on-chip tile memory; and determining a display pixel coloring value for each of a plurality of display pixels overlapping the rendered pixel at a target resolution greater than the first resolution based on the coloring values ​​of the sample set, and storing an upscaled representation of the rendered pixel in the on-chip tile memory.

[0017] Both rendering (including pixel shading) and upscaling can be completed in a single render pass along the GPU pipeline. This can produce a high-quality final image at high resolution. In some implementations, there may be no sharp object edges, resulting in very low rendering costs and low memory bandwidth consumption at low shading resolutions on mobile GPUs.

[0018] Shading can be performed once per rendered pixel. Using per-pixel shading at low resolutions can produce high-quality final images at high resolutions with low rendering costs.

[0019] The target resolution can be larger than the first resolution by a factor equal to the sampling density of the rendered pixels. This allows for reading samples of each pixel from on-chip memory during upscaling in the on-chip tile memory of the mobile GPU and resolving them one by one to the corresponding display pixels (one sample per display pixel).

[0020] The sampling density can correspond to the multisample anti-aliasing level. The multisample anti-aliasing (MSAA) level can be 4x or 16x. Other MSAA levels can also be used. This allows pixels to be rendered at a lower resolution before being scaled up to the target resolution.

[0021] This system can be implemented using a mobile graphics processing unit. This allows the system to be implemented in consumer products such as smartphones and tablets.

[0022] Rendering a pixel can further include, for each of the sample locations, determining whether that location is covered by primitives in the image. The shading value of each sample at a location determined to be covered by primitives can be equal to the shading result of the rendered pixel. This can lead to lower rendering costs.

[0023] The shading location of a rendered pixel can be located at the centroid of a sample point determined to be covered by primitives in the image. Each of the samples determined to be covered by primitives in the rendered pixel can correspond to the same color in the rendered pixel, which is generated by per-pixel shading. Therefore, a pixel can be shading only once, and the result is only assigned to the samples covered by primitives.

[0024] The processor can be configured to subsequently write the upscaled representation stored in the on-chip tile memory to the frame buffer in system memory. This allows the image to be displayed subsequently.

[0025] The system can be configured to implement an application programming interface (API). The API can be Vulkan or OpenGLES. Other suitable APIs can also be used. Sub-pass features (or equivalent features) of APIs such as Vulkan can be used to perform custom resolution (from pixels with multiple samples to multiple final pixels) within the on-chip tile memory.

[0026] Using extensions in these APIs allows sample locations to be moved to regular grid locations within pixels. In Vulkan, a rendering pass can be split into multiple sub-passes, allowing rendering and upscaling to be performed in a single processing pass without requiring modifications to the GPU hardware.

[0027] The step of storing the upscaled representation may include converting one of the samples of the rendered pixel and / or one of the shading values ​​of the samples into a display pixel at the target resolution. This can allow pixels to be rendered at a low resolution before upscaling to a high resolution.

[0028] The target resolution can be 4K or 8K. This allows the system to be implemented in current high-resolution consumer electronics. Other target resolutions can also be used.

[0029] According to a second aspect, a method is provided for processing an image in an image processing system, the image comprising a plurality of rendered pixels, the system being configured to divide the image into a plurality of tiles, each tile comprising a subset of the rendered pixels of the image and corresponding to a plurality of display pixels, the method comprising, in a processing pass, for at least one rendered pixel in the image: rendering the rendered pixel by: determining a set of sample points, each sample point having a respective position in the rendered pixel and these positions having a common sampling density in the rendered pixel; coloring the rendered pixel at a first resolution to give a coloring result of the rendered pixel; and writing a coloring value of each of the sample points into an on-chip tile memory based on the coloring result; and storing an upscaled representation of the rendered pixel in the on-chip tile memory by determining, based on the coloring values ​​of the sample point set, a display pixel coloring value of each of a plurality of display pixels overlapping the rendered pixel at a target resolution greater than the first resolution.

[0030] Using this method, both rendering and scaling can be completed in a single render pass along the GPU pipeline. This can produce a high-quality final image at high resolution. In some implementations, there may be no sharp object edges, and very low rendering costs on mobile GPUs (due to per-pixel shading at low resolutions) and low memory bandwidth consumption.

[0031] The method may further include: for each of the sample locations, determining whether the corresponding sample location is covered by primitives in the image. The shading value of each sample point with a location determined to be covered by primitives can be equal to the shading result of the rendered pixel. This may result in lower rendering costs.

[0032] The target resolution can be greater than the first resolution by a factor equal to the sampling density of the rendered pixels. This allows the shading value of each pixel sample to be read from the on-chip memory and parsed one by one into the corresponding display pixel in the on-chip tile memory (one shading value per display pixel).

[0033] The sampling density can correspond to the level of multisampling anti-aliasing. This allows pixels to be rendered at a lower resolution before being scaled up to the target resolution.

[0034] According to a third aspect, a computer program is provided that, when executed by a computer, causes the computer to perform the methods described above. The computer program may be provided on a non-transitory computer-readable storage medium. Attached Figure Description

[0035] This application will now be described by way of example with reference to the accompanying drawings.

[0036] In the attached diagram:

[0037] Figure 1 It shows the requirements for different screen resolutions.

[0038] Figure 2 This illustration schematically shows rendering at a reduced resolution, followed by upscaling.

[0039] Figure 3 The diagram illustrates a pipeline in which both rendering and upscaling are completed in a single rendering pass.

[0040] Figure 4 (a) and Figure 4 (b) schematically illustrates the rendering with and without centroid sampling.

[0041] Figure 5 This schematically illustrates moving the sample point location to a regular grid location within a pixel.

[0042] Figure 6 This illustrates the modification of the GPU's tile cache hardware block before output to system memory.

[0043] Figure 7 A flowchart illustrating an example of a method for processing images is shown.

[0044] Figure 8 An example of a graphics processing system is shown.

[0045] Specific implementation method

[0046] In this article, rendering refers to any form of generating a visual image, such as for displaying an image on a computer screen, printing, or projecting.

[0047] Figure 3 A rendering pipeline implemented by a graphics processing system is illustrated. The system implementing the pipeline includes an on-chip tile memory and a processor. The system implements tile-based rendering and is configured to receive an image to be rendered, comprising a plurality of renderable pixels. The system is configured to divide the image into a plurality of tiles, each tile comprising a subset of the renderable pixels of the input image and corresponding to a plurality of display pixels.

[0048] As will now be described, the processor is configured to perform image rendering and upscaling in a single pass along the mobile GPU pipeline.

[0049] During the rendering phase, after vertex shading as shown in 301, the image is rasterized, as shown in 302. Multisample anti-aliasing (MSAA) is used to mitigate jagged edges of the geometry. This is facilitated by using a multisample cache (render target). Some possible configurations for the cache are 2x, 4x, 8x, 16x (i.e., nx, where "n" represents the number of samples allocated per pixel). MSAA occurs as part of the rasterization phase 302 of the pipeline, and... Figure 3 It is usually indicated as 303. Figure 3 The example shown uses 16x MSAA. However, other MSAA levels, such as 4x, can also be used.

[0050] MSAA, supported by the GPU in hardware, allows for shading of a pixel only once (i.e., calculating the RGB color in the pixel shader) for a primitive, but storing the result across multiple samples for that pixel. In one implementation, such as... Figure 3 Primitives like triangle 304 can be tested over each of "n" sample points to create a 16-bit (for 16x MSAA) overlay mask identifying the pixel portions covered by the triangles. A pixel shader 305 is then executed once, and the pixel's shading result is assigned to the sample points identified by the overlay mask. The fixed-function GPU hardware then resolves this multisampling buffer into a single color value for each final pixel in the framebuffer to address aliasing.

[0051] By default, the parsing hardware can calculate the average color of these multiple samples as the final color for each pixel. For example, if the first three samples of a pixel contain the exact same shading (at 4x MSAA), and the fourth sample contains the background color, averaging them will produce an anti-aliased output for the last pixel on the object's edge. However, as will be described in more detail below, in the system described herein, centroid sampling can be used, and for primitives, pixels can be shading once, but the result is only assigned to the samples covered by the primitive.

[0052] Therefore, using MSAA, the pipeline determines a set of n samples per pixel, each sample having its own location within the pixel, and these locations collectively having a sampling density equal to the MSAA level within the pixel. In one implementation, it can then be determined which samples in the pixel are covered by primitives in the image.

[0053] Then, pixel shader 305 is executed to shade each rendered pixel once (pixel-by-pixel shading) to provide a shading result for each pixel. Then, based on the shading result of the pixel, the shading value of each of the samples is written to the on-chip tile memory (i.e., the tile cache). In one implementation, the shading result is assigned to samples determined by the system to be covered by primitives. After rendering, as shown in 306, a custom resolution is then performed on the multiple samples of the rendered pixel stored in the on-chip tile memory (tile cache). As will be described in more detail below, after determining the display pixel shading value for each display pixel overlapping the rendered pixel based on the shading values ​​of the sample set of the rendered pixel, an upscaled representation of the rendered pixel is stored in the on-chip tile memory. One of the shading values ​​from this sample set is assigned to a display pixel as its shading value in the high-resolution image.

[0054] The processor can be configured to subsequently write the upscaled representation to the frame buffer in system memory at the target resolution. Once all pixels in the image have been rendered, upscaled, and output to system memory, the final image 307 can be displayed.

[0055] Therefore, both image rendering (including pixel shading) and upscaling are performed in a single rendering pass along the pipeline. This pipeline supports rendering at low resolution (LR) while obtaining the final image at high resolution (HR) in a single rendering pass on the mobile GPU.

[0056] This is achieved by implicitly utilizing the hardware's MSAA functionality. Upscaling can be enabled to upscale the fragment output (from the pixel shader) via a custom resolution operation that resolves the MSAA sample colors to the final pixel colors within the mobile GPU's on-chip tile memory. As a result, the intermediate MSAA cache does not need to access system memory during upscaling.

[0057] Therefore, after all other fragment operations (including, for example, alpha blending) are completed, each of the multiple samples of the rendered pixel is custom-parsed to produce a single display pixel color value, which can then be written to the corresponding color cache in system memory under HR.

[0058] In one implementation, the writing of the color cache can be deferred to a later time, after all pixels of the tile have been rendered and upscaled, and can be output to system memory in batches corresponding to the tiles.

[0059] In a specific example, the algorithm steps for 4x scaling (corresponding to rendering using 4x MSAA level) are as follows. With 4x MSAA enabled, 3D rendering is performed to the low-resolution framebuffer. Per-pixel shading is performed at the low resolution, that is, each pixel (with four samples) is shading once, so that all samples within the pixel covered by primitives will have the same shared color.

[0060] In one implementation described in this paper, the pixel's shading location is located at the geometric center of all covered sample locations of the fragment. Figure 4 (a) and Figure 4 (b) schematically illustrates the rendering with and without centroid sampling. Figure 4 In (a), the shading position 401 of pixel 402 is located at the center of all sample points 403, 404, 405, and 406, regardless of whether these sample points are covered by primitive 407. Figure 4 In (b), using centroid sampling, the shading position 408 of pixel 402 is located only at the center of the covered samples 404 and 406. Therefore, when using centroid sampling, a pixel is shading once, and the result is only assigned to the samples covered by the primitive. This can be achieved by adding the "centroid" shader qualifier to all fragment shader inputs (for variable interpolation).

[0061] As described above, at the end of the tile-based rendering process, just before the tile buffer is written to system memory (i.e., the frame buffer in DDR), a custom resolution algorithm within the on-chip tile buffer is used to perform custom resolution. Using 4xMSAA, four samples of each pixel are read from the LR tile buffer and resolved one by one into 2x2 display pixels (one sample per display pixel) in the HR tile buffer, which can then be written to the HR frame buffer in system memory for display. The result is an HR frame buffer that is four times larger than the LR frame buffer.

[0062] GPUs can implement application programming interfaces such as Vulkan or OpenGL ES.

[0063] When the API is OpenGL ES, the API extension GL_ARB_sample_location (global user-defined sample location) can be used to move all sample locations to regular grid locations within pixels (e.g., ...). Figure 5 (as shown in (b)) instead of using the hardware default location, such as Figure 5 As shown in (a) in the figure. Figure 5 (a) and Figure 5 The positions shown in (b) correspond to the global settings for all pixels in the frame buffer.

[0064] Therefore, the sample point position can be changed from the default position to a regular grid position within the rendered pixel. This occurs during the rendering step prior to the upscaling step.

[0065] When using the Vulkan API, you can similarly use the corresponding API extension VK_EXT_sample_locations to move sample locations to regular grid locations.

[0066] In another example, the algorithm steps for scaling at 16x are as follows. Rendering to the LR framebuffer is performed with 16x MSAA enabled. Per-pixel shading is performed under LR, shading one pixel (with 16 samples) once, so that all samples within the pixel covered by primitives will have the same color. Similar to... Figure 5 In the example shown in (b), by using the "centroid" shader qualifier for all fragment shader inputs, the shading position of the fragment is moved to the geometric center of the position of all sample points covered by the primitive (for variation interpolation).

[0067] Custom parsing is performed at the end of tile-based rendering (before the tile buffer is written to DDR). 16 samples from each pixel in the LR tile buffer are parsed one by one into 4x4 display pixels in the final HR tile buffer (only one sample per display pixel), which are then written to the HR frame buffer in DDR for display. The result is an HR frame buffer, which is 16x larger than the LR frame buffer.

[0068] The API extension GL_ARB_sample_location for OpenGL ES and VK_EXT_sample_locations (global user-defined sample locations) for Vulkan can be used to move sample locations to regular grid locations within a pixel, such as... Figure 5 As shown in (b) of the diagram, 501b, 502b, 503b, and 504b are used instead of... Figure 5 The default positions 501a, 502a, 503a, and 504a are shown in (a) in the figure.

[0069] A custom parsing algorithm can be implemented at the pixel level of Lightroom (each pixel contains multiple samples, and each sample can have 4 bytes of RGBA color) using the following pseudocode:

[0070]

[0071] Two exemplary solutions for implementing custom parsing algorithms on mobile GPUs will now be described. These solutions conveniently utilize the on-chip block memory of the mobile GPU. One solution is via dedicated fixed-function hardware. The other solution is via the subpass feature in the Vulkan API. Both solutions require only a single rendering pass from the GPU to perform both rendering and upscaling.

[0072] In the first solution, such as Figure 6 As shown, hardware modifications are required to resolve LR attachments to HR attachments. Dedicated fixed-function hardware implements custom resolution at the local tile memory 601, which only accesses on-chip memory for resolution.

[0073] The local tile memory (on-chip tile memory) 601 on the mobile GPU 602 is modified as described below. For rendering tiles, the tile cache module maintains two on-chip color caches (for tiles) at two different resolutions: LR and HR. The LR cache requires a high MSAA level (e.g., 4x or 16x), while the HR cache requires only one sample per pixel.

[0074] like Figure 6 As shown, the local tile memory 601 performs a custom parsing at the end of the tile-based rendering pipeline (before the tile cache is written to DDR), such that the color of each sample at the LR cache can be written to the HR tile cache as the final pixel color, which will eventually be written to the explicit frame buffer 603 in DDR 604 for display.

[0075] Since the two tile buffers have the same memory size (although one is in LR and the other in HR, and they have different MSAA levels), in-situ resolution can be used by allocating only one copy of the tile buffer. For example, the MSAA level of the LR tile buffer can be 4 or 16, while the MSAA level of the HR tile buffer can be 1, with only one sample per pixel.

[0076] For scaling up to 4x, this in-situ resolution can be achieved by reading the RGBA values ​​of multiple (4x) samples of a pixel and then writing them as a 2x2 pixel block to the same address in the tile buffer. Since the RGBA values ​​of multiple (4x) samples of the pixel are no longer needed after reading, the same memory location can be overwritten as an HR buffer.

[0077] In the second solution, custom parsing can be performed without hardware modification by using the sub-pass feature of the Vulkan API. This implementation requires no changes to the GPU driver or hardware.

[0078] For tile-based renderers in mobile GPUs, if subsequent rendering operations are at the same resolution and require data from the same pixel location currently being rendered (accessing different pixel locations would require accessing values ​​outside the current tile, breaking this optimization), the results of previous renderings can be effectively preserved on-chip. Vulkan's solution is to split the rendering operations of a single rendering pass into multiple sub-passes that share the same resolution and tile arrangement, allowing a sub-pass to access the results of previous sub-passes.

[0079] Therefore, in Vulkan, a rendering pass can include multiple sub-passes, and a sub-pass can access the rendering results of previous sub-passes of tiles that are still stored in the on-chip memory of the mobile GPU before being output to system memory.

[0080] This solution leverages this feature to achieve a single-pass pipeline by using two sub-passes, including both rendering and custom parsing.

[0081] It should be noted that in Vulkan, multiple sub-steps of a single iteration share the same unique rendering resolution, and render targets with multiple different resolutions (e.g., LR and HR) cannot be output. To maintain consistency with the Vulkan specification, the render targets (color attachments) of both sub-steps can be set to LR. However, only the render target of the first sub-step is likely to be actually used, and the second sub-step may not output its render target. Instead, the second sub-step can output a separate stored image in HR (e.g., using the imageStore() function in the GLSL shading language), allowing the system to output arbitrary resolutions at arbitrary pixel locations.

[0082] Therefore, this solution uses the stored image as output to compensate for sub-pass rendering so that the system can output a resolution higher than the target resolution.

[0083] The algorithm steps for scaling at 4x are as follows. In the first sub-pass: with 4x MSAA enabled, pixels are rendered to their render target in LR. The second sub-pass performs a custom resolution, which renders a full-screen quad. In the fragment shader, four samples are read from the render target of the first sub-pass (as input) for each pixel, but the result of the resolution (i.e., 4 output pixels) is written to the stored image in HR (i.e., using the imageStore function), instead of to its own render target. Therefore, the second sub-pass may not output anything to its own render target. Although its render target is set in LR, it may not be used. An example of the input and output is as follows:

[0084] Input: subpassLoad(gsubpassInputMS subpass0,int sample);

[0085] Output: imageStore to storage_image, created using VK_IMAGE_USAGE_STORAGE_BIT, using this shader function: imageStore(gimage2D image,ivec2 P,gvec4 data);

[0086] In one example, the shader code for the second sub-cycle could look like this:

[0087]

[0088]

[0089] The advantage of the second exemplary solution described above is that it does not require modification of the driver or hardware. As shown in the table below, the pipeline can also support other upscaling factors, where MSAA_level = xscale * yscale, and (xscale, yscale) are upscaling value pairs along the X and Y directions.

[0090] Table 1

[0091] MSAA_level Possible scaling value pairs: (xscale,yscale) 2x (1, 2) or (2, 1) 4x (2, 2), (1, 4) or (4, 1) 8x (2, 4), (4, 2), (1, 8) or (8, 1) 16x (4, 4), (2, 8), (8, 2), (1, 16) or (16, 1)

[0092] If the scaling value pair is not (1, 1), the pixel color value is rendered to the intermediate render buffer in LR with an MSAA level greater than 1x, such as 2x, 4x, 8x, or 16x. The resulting final image resolution in HR is 1 / 2 times the MSAA_level of LR.

[0093] The solution described above using the sub-pass feature can also be implemented using other similar 3D APIs, such as DirectX 12 or Metel. Any API with a sub-pass feature can be used, whereby the first sub-pass can be used to render pixels, and the second sub-pass can be used to scale pixels to the target resolution. For the OpenGL ES API, the sub-pass solution can be similarly implemented in the GPU driver by an end-of-tile shader (also known as a post-frame shader), which is a fragment shader that post-processes the pixels of the current tile. This can be started immediately after the tile is rendered to the tile buffer.

[0094] In the example described above, MSAA was used to determine the sample set for each rendered pixel. However, supersampling anti-aliasing (SSAA) also operates based on the same principles and can be used alternatively. SSAA performs a pixel shader on all covered samples for each pixel. Therefore, SSAA can be more expensive than MSAA. MSAA operates along the edges of geometry, while SSAA operates even inside the geometry. This results in higher visual quality, but at a higher performance cost compared to MSAA. For this reason, MSAA can be used.

[0095] Figure 7 A flowchart illustrating an example of a method for processing an image in an image processing system, the image comprising a plurality of rendered pixels, the system being configured to divide the image into a plurality of tiles, each tile comprising a subset of the rendered pixels of the image and corresponding to a plurality of display pixels. The method includes the following steps for at least one rendered pixel in the image in a processing pass. In steps 701 to 703, the rendered pixel is rendered. In step 701, the method includes determining a set of sample points, each sample point having a respective location within the rendered pixel, and these locations sharing a common sampling density within the rendered pixel. In step 702, the method includes coloring the rendered pixel at a first resolution to give a coloring result for the rendered pixel. In step 703, the method includes writing a coloring value for each of the sample points into an on-chip tile memory based on the coloring result. In step 704, the rendered pixel is upscaled by storing an upscaled representation of the rendered pixel in the on-chip tile memory to determine display pixel coloring values ​​for each of a plurality of display pixels overlapping the rendered pixel at a target resolution greater than the first resolution, based on the coloring values ​​of the sample point set.

[0096] Figure 8 This is a schematic diagram of a system 800 configured to perform the methods described herein. System 800 can be implemented on a device such as a laptop, tablet, smartphone, television, or any other device in which graphics data is to be processed. System 800 is implemented, for example, by a mobile GPU.

[0097] System 800 includes a graphics processor 801 configured to process data. For example, processor 801 may be a GPU. In one embodiment, processor 801 may be implemented as a computer program running on a programmable device, such as a GPU or a central processing unit (CPU). System 800 includes on-chip memory 802 configured to communicate with graphics processor 801. System 800 may include more than one processor and more than one memory. The memory may store data executable by the processor. The processor may be configured to operate according to a computer program stored in a non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its methods in the manner described herein. The processor may also write data to external system memory (…). Figure 8 (Not shown in the image).

[0098] In summary, in the GPU described herein, rendering is performed under LR using MSAA, and a custom resolution is performed at the tile cache (on-chip memory of the GPU) to produce the HR image. Pixel shading is performed under LR, shading each pixel (with n samples) once. In one implementation, all samples covered by primitives within a pixel will have the same color. The custom resolution is performed at the tile cache before the upscaled samples are written from the tile cache to system memory. For example, with 16x MSAA, 16 samples from the rendered pixel in LR are output to 4×4 pixels in the resulting HR image (one sample per pixel). Therefore, instead of using the hardware's default behavior of MSAA resolution in tile-based mobile GPUs, instead of using MSAA resolution with MSAA enabled, the pixel is rendered under LR, and then a custom resolution operation is performed at the end of the same rendering pipeline to produce the final HR (high resolution) image.

[0099] Therefore, the system is configured to perform rendering (including pixel shading) and upscaling in a single rendering pass on the mobile GPU. In some implementations, this can result in significantly less memory and bandwidth consumption compared to a two-pass approach (i.e., one pass for rendering and one pass for upscaling).

[0100] This enables pixel shading at low resolutions with very low shading costs, and produces high-resolution final images without sharp object edges. This is achieved by implicitly utilizing hardware MSAA capabilities and by using a custom resolution solution at the end of the same rendering pipeline.

[0101] This could be useful for various rendering techniques that aim to generate high-quality upscaled images for large-screen products in an efficient and inexpensive manner, without sharp edges.

[0102] While LR may have significantly lower fragment shading costs, it can achieve the same final HR image. Generating an HR image from an LR image does not require explicit stretching. In some implementations, sharp edges may be absent due to 4x or 16x MSAA rasterization. The edges of polygons (the most noticeable source of aliasing in 3D graphics) can be anti-aliased.

[0103] In some implementations, the method described in this paper can achieve higher bandwidth efficiency because the tile-based rendering pipeline utilizes on-chip tile memory on the mobile GPU. This allows custom parsing to access only the on-chip tile memory used for intermediate rendering buffers (rather than system memory) to produce the final HR image, requiring only a single rendering pass from the processor without any image stretching from LR to HR. Therefore, this method can be significantly more memory-bandwidth efficient and thus more suitable for bandwidth-constrained mobile GPUs than traditional methods.

[0104] Lower GPU fragment shading rates (in LR) and lower memory and bandwidth consumption can extend the battery life of mobile devices and improve the frame rate of rendering complex and demanding games.

[0105] The applicant therefore separately discloses each individual feature described herein, as well as any combination of two or more such features, to a certain extent that such features or combinations can be implemented based on the specification as a whole, according to common general knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein, and without limiting the scope of the claims. The applicant notes that aspects of this application may consist of any such individual feature or combination of features. In view of the foregoing description, it will be apparent to those skilled in the art that various modifications can be made within the scope of this application.

Claims

1. A graphics processing system configured to process an image comprising a plurality of rendered pixels, the system being configured to divide the image into a plurality of tiles, each tile comprising a subset of the rendered pixels of the image and corresponding to a plurality of display pixels, the system comprising: On-chip image block memory; as well as The processor is characterized in that it is configured to perform a processing pass for at least one rendered pixel in the image, comprising: The rendered pixels are rendered in the following manner: Determine a set of sample points, where each sample point has its own location in the rendered pixel, and these locations have a common sampling density in the rendered pixel; The rendered pixels are colored at a first resolution to give the coloring result of the rendered pixels; and Based on the coloring result, the coloring value of each of the sample points is written into the on-chip tile memory; and Performing custom parsing within the on-chip tile memory includes: determining the display pixel shading value of each of a plurality of display pixels that overlap with the rendered pixel at a target resolution greater than the first resolution, based on the shading value of the sample set; and storing the up-scaled representation of the rendered pixel in the on-chip tile memory. The rendering of the rendering pixel further includes determining whether each sample point position is covered by a primitive in the image, and the shading position of the rendering pixel is located at the centroid of the sample point that is determined to be covered by a primitive in the image.

2. The system according to claim 1, characterized in that, Shading is performed once per rendered pixel.

3. The system according to claim 1, characterized in that, The target resolution is larger than the first resolution, such that during upscaling in the on-chip tile memory, the samples of the rendered pixels are read from the on-chip tile memory and parsed one by one into the corresponding display pixels.

4. The system according to any one of claims 1 to 3, characterized in that, The sampling density corresponds to the multisampling anti-aliasing level.

5. The system according to claim 4, characterized in that, The multisampling anti-aliasing level is 4x or 16x.

6. The system according to any one of claims 1 to 3, characterized in that, The system is implemented by a mobile graphics processing unit.

7. The system according to any one of claims 1 to 3, characterized in that, The shading value of each sample point at the location determined to be covered by the primitive is equal to the shading result of the rendered pixel.

8. The system according to claim 1, characterized in that, Each of the samples identified as being covered by primitives in the rendered pixel corresponds to the same color in the rendered pixel.

9. The system according to any one of claims 1 to 3, characterized in that, The processor is configured to subsequently write the upscaled representation stored in the on-chip tile memory into the frame buffer in the system memory.

10. The system according to any one of claims 1 to 3, characterized in that, The system is configured to implement an application programming interface, wherein the application programming interface is Vulkan or OpenGL ES.

11. The system according to any one of claims 1 to 3, characterized in that, The step of storing the upscaled representation includes converting one of the samples of the rendered pixel and / or one of the shading values ​​of the samples into a display pixel at the target resolution.

12. The system according to any one of claims 1 to 3, characterized in that, The target resolution is 4K or 8K.

13. A method for processing an image in an image processing system, the image comprising a plurality of rendered pixels, the system being configured to divide the image into a plurality of tiles, each tile comprising a subset of the rendered pixels of the image and corresponding to a plurality of display pixels, the method, for at least one rendered pixel in the image, comprising the following steps in a processing pass: The rendered pixels are rendered in the following manner: Determine a set of sample points, where each sample point has its own location in the rendered pixel, and these locations have a common sampling density in the rendered pixel; The rendered pixels are colored at a first resolution to give the coloring result of the rendered pixels; as well as Based on the coloring result, the coloring value of each of the sample points is written into the on-chip tile memory; as well as Performing custom parsing within the on-chip tile memory includes: determining the display pixel shading value of each of a plurality of display pixels that overlap with the rendered pixel at a target resolution greater than the first resolution, based on the shading value of the sample set; and storing the up-scaled representation of the rendered pixel in the on-chip tile memory. The rendering of the rendering pixel further includes determining whether each sample point position is covered by a primitive in the image, and the shading position of the rendering pixel is located at the centroid of the sample point that is determined to be covered by a primitive in the image.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a computer, it causes the computer to perform the method of claim 13.

Images