Upscaling based on Multi-Sample Anti-Aliasing (MSAA)

Abstract

A processing circuit (e.g., a GPU) obtains a first MSAA intermediate buffer from rendering a first image, the first MSAA intermediate buffer containing multiple subsamples of each pixel in the first image. For a first pixel in the first image, the processing circuit determines a first surrounding subsample pattern based on a first subsample in the surrounding region of the first pixel. Furthermore, based on the first surrounding subsample pattern, the processing circuit obtains a first blend weight vector for a first position among multiple positions that define the output pixel positions within the first pixel for upscaling the first pixel to multiple output pixels in an upscaled image with a higher resolution than the first image. The first blend weight vector contains a first blend weight for each of the first candidate subsamples associated with the first pixel. The processing circuit generates the upscaled image accordingly.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the benefit of priority based on U.S. Patent Application No. 18 / 386,141, filed on November 1, 2023, the entire content of which is incorporated herein by reference.

[0002] [Technical Field] This disclosure generally relates to graphics processing, and more particularly, to embodiments related to rendering techniques in graphics processing.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The research of the currently named inventors within the scope described in this background art section, as well as aspects of the description that may not be eligible as prior art at the time of filing, are not admitted as prior art to the present disclosure, either explicitly or implicitly.

[0004] In graphics processing, rendering can refer to the process of generating an image from a three - dimensional (3D) model or a two - dimensional (2D) model. In some examples, a computing system can include a graphics processing unit (GPU) to accelerate rendering, and the rendering process in a computing system can sometimes be referred to as a graphics pipeline. The graphics pipeline can include various operations such as vertex shading operations, fragment shading operations, and the like.

Summary of the Invention

Means for Solving the Problems

[0005] Aspects of this disclosure provide methods and apparatus for upscaling based on multi-sample anti-aliasing (MSAA). In some examples, a processing circuit (e.g., a GPU) obtains a first multi-sample anti-aliasing (MSAA) intermediate buffer from rendering a first image, the first MSAA intermediate buffer containing a plurality of subsamples for each pixel of the first image. For a first pixel of the first image, the processing circuit determines a first surrounding subsample pattern based on a first subsample in the surrounding region of the first pixel. Furthermore, based on the first surrounding subsample pattern, the processing circuit obtains a first blend weight vector for a first position among a plurality of positions that define output pixel positions within the first pixel for upscaling the first pixel to a plurality of output pixels in an upscaled image with a higher resolution than the first image. The first blend weight vector contains a first blend weight for each of the first candidate subsamples associated with the first pixel. The processing circuit generates an upscaled image corresponding to the first image, and the value for the first output pixel of the upscaled image at the first position within the first pixel of the first image is calculated as a weighted sum of first candidate subsamples according to the first blend weights.

[0006] In some examples, rendering a first image generates multiple subsamples for each pixel of the first image, each having a Rotated Grid Supersampling (RGSS) pattern. In one example, rendering an image generates four subsamples for each pixel of the first image, each having a 4xRGSS pattern.

[0007] In some examples, the first subsample in the region surrounding the first pixel includes multiple subsamples in the first pixel. In one example, the first subsample in the region surrounding the first pixel includes one or more subsamples in one or more neighboring pixels of the first pixel.

[0008] In some examples, the first ambient subsample pattern is a subsample difference pattern. The processing circuit determines the difference between two subsamples in the first subsample and determines the bits in the first ambient subsample pattern based on the difference. In one example, the difference is at least one of chrominance, luminance, and depth differences.

[0009] In some examples, the first subsample used to generate the first surrounding subsample pattern is from the same set of subsamples as the first candidate subsample.

[0010] In some examples, the first subsample used to generate the first surrounding subsample pattern includes the first candidate subsample.

[0011] In some examples, the first candidate subsample includes the first subsample used to generate the first surrounding subsample pattern.

[0012] In some examples, the processing circuit determines an index to a lookup table according to a first ambient subsample pattern, and the lookup table contains a set of blend weight vectors indexed according to the ambient subsample pattern.

[0013] In some examples, the processing circuit pre-trains a set of blend weight vectors for the surrounding subsample pattern by using at least one reference image with a resolution greater than or equal to that of the upscaled image. For example, the processing circuit obtains a second MSAA intermediate buffer from rendering a second image with the same camera settings as the reference image, and the second image has the same resolution as the first image. Based on the second MSAA intermediate buffer and the reference image, the processing circuit collects votes for candidate subsamples for the surrounding subsample pattern and determines the blend weights in the blend weight vector associated with the surrounding subsample pattern according to the votes of the candidate subsamples. In one example, for the position of a pixel in the second image that has a corresponding pixel in the reference image, the processing circuit determines a first candidate subsample from the candidate subsamples that has the smallest error with respect to the corresponding pixel in the reference image and adds a vote to the first candidate subsample.

[0014] Aspect of this disclosure also provides a non-temporary computer-readable medium for storing instructions that cause a computer to perform a method for upscaling based on multi-sample anti-aliasing (MSAA) when executed by the computer.

[0015] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description and accompanying drawings below. [Brief explanation of the drawing]

[0016] [Figure 1] Block diagrams of electronic devices according to some embodiments of this disclosure are shown.

[0017] [Figure 2] The diagrams show pixels with sampling patterns in several examples.

[0018] [Figure 3]FIG. illustrates a technique for determining sub-sample difference patterns in some examples.

[0019] [Figure 4] FIG. shows the votes of the most matching candidates in one example.

[0020] [Figure 5] FIG. shows the downsampling of blend weights in some examples.

[0021] [Figure 6] FIG. shows runtime upscaling in some examples.

[0022] [Figure 7] FIG. shows a flowchart outlining a process according to an embodiment of the present disclosure.

[0023] [Figure 8] FIG. shows a flowchart outlining a process according to some embodiments of the present disclosure.

[0024] [Figure 9] FIG. shows a flowchart outlining a process according to some embodiments of the present disclosure.

[0025] [Figure 10] FIG. is a schematic diagram of a computer system according to an embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION

[0026] The detailed description provided below with respect to the accompanying drawings is intended to explain various configurations and is not intended to represent the only configuration in which the concepts described herein can be implemented. The detailed description includes specific details for the purpose of providing a complete understanding of the various concepts. However, these concepts can be implemented without these specific details.

[0027] In this disclosure, some technical terms are used with the following definitions.

[0028] FPAA stands for full-scene anti-aliasing, which conceptually works by simply rendering the scene at a higher resolution and then downsampling it to a lower resolution output.

[0029] MSAA stands for multi-sample anti-aliasing and refers to a specific optimization of supersampling (or supersampling). For example, a renderer evaluates the fragment program once per pixel and supersamples depth and stencil values. The term multisampling (or multisampling) refers to any special case of supersampling in which some components of the final image are not fully supersampled. For example, in the case of full N×N supersampling, a pixel is sampled at N×N points within the pixel, where the N×N points form a grid and are spaced apart along the axes, where N is a positive integer; in the case of N×multisampling, the number of sampling points within a pixel is N, where the N points are sparsely distributed and not all grids have sampling points.

[0030] RGSS stands for rotated grid supersampling, a technique used in computer graphics and rendering to improve the quality of images produced by rendering algorithms, particularly in the context of anti-aliasing. Anti-aliasing is the process of reducing jagged or "jagged" edges (aliasing artifacts) that can occur in computer-generated images, especially when rendering diagonal lines or curves. Regular grid-based supersampling takes multiple samples within each pixel at regular intervals along axes to estimate the pixel's color and other properties. While regular grid-based supersampling is helpful in reducing aliasing artifacts, it may not be as effective when dealing with rotated or diagonal lines. Rotated grid supersampling addresses this problem by using a rotated grid pattern for sampling.

[0031] Super-resolution (SR) is a technique used in image processing and computer vision to improve the resolution or level of detail of an image or video beyond its original quality. The goal of super-resolution is to generate a high-resolution version of an image or video from one or more low-resolution source images. This technique is particularly useful in applications where higher quality images or videos are required, such as medical imaging, surveillance, satellite imagery, and improving the quality of old or low-resolution photographs.

[0032] Single-image super resolution (SISR) is the process of increasing the resolution of a single low-resolution image. SISR can be achieved by learning a mapping function from low-resolution images to high-resolution images using techniques such as deep learning, including convolutional neural networks (CNNs).

[0033] Multi-image super resolution (MISR) technology generates a higher-resolution image using multiple low-resolution images of the same scene or object, taken from slightly different viewpoints or with small differences in motion. MISR leverages information across multiple images to improve resolution.

[0034] A fragment refers to a collection of values ​​generated by a rasterizer. Each fragment represents a segment of the sample size of a rasterized primitive. In some examples, the size covered by a fragment relates to the pixel area. In some examples, rasterization can generate multiple fragments per pixel from the same triangular primitive, depending on various multisampling parameters and rendering conditions. In some examples, at least one fragment is generated for each pixel area covered by the primitive being rasterized. In some examples, a fragment corresponds to a basic unit generated by rasterization and may contain a set of colors and a single depth value. In one example, a fragment corresponds to a pixel. In other examples, a fragment corresponds to a sample when multiple samples are generated within the area of ​​a pixel. To make the explanation clearer and easier to understand, when multiple samples are generated within the area of ​​a pixel, the samples are referred to as subsamples, as sub-pixel samples, and fragments correspond to subsamples.

[0035] Alpha testing is a technique used in rendering to determine whether a pixel in a rendered image should be displayed based on its alpha (transparency) value. Alpha testing is primarily used in computer graphics and 2D or 3D rendering to handle transparency in textures or images.

[0036] According to one aspect of this disclosure, rendering high-resolution images requires significant computational resources, memory bandwidth, and storage space. High-resolution image output is particularly challenging on low-power devices such as mobile phones, tablets, and portable game consoles. Several aspects of this disclosure provide efficient upscaling techniques that output high-resolution images based on low-resolution intermediate images.

[0037] Generally, resolution upscaling methods are complex and may require special hardware or high runtime computing resources to produce high-quality results, which may not be suitable for execution on low-power devices. On the other hand, simple algorithms with low computational costs often fail in terms of output quality.

[0038] In the first relevant example, special acceleration hardware called TensorCore can operate on trained neural networks and multi-image super-resolution (MISR) to obtain high-quality results. While the quality is high, it requires dedicated hardware to accelerate the algorithm and is therefore not applicable to general-purpose graphics acceleration hardware. Furthermore, a high-end desktop processor is required to achieve good performance.

[0039] In a second related example, open-source super-resolution solutions do not require dedicated hardware and can operate based on single-image super-resolution (SISR) mode or multi-image super-resolution (MISR) mode. However, open-source super-resolution solutions still need to run on high-performance desktop GPUs, and the complexity remains very high on low-power handheld devices. Furthermore, the SISR portion is based on conventional low-resolution images using conventional image processing algorithms such as sharpening filters. As a result, in this second related example, open-source super-resolution solutions lack knowledge of high-resolution geometric information and may fail to reconstruct proper high-resolution edges from jagged low-resolution inputs.

[0040] Several aspects of this disclosure provide a technique that uses the intermediate results of multi-sample anti-aliasing (MSAA) for super-resolution upscaling of a single image. This technique is referred to as MSAA-based upscaling.

[0041] Generally, GPU hardware units are configured to support multi-sample anti-aliasing (MSAA), even in GPUs in low-power handheld devices. For example, when a GPU renders an output image, MSAA can generate microgeometric information, such as the geometry information of subpixels in the output image, as an intermediate result, and then generate pixels in the output image based on this intermediate result. The subpixel geometry information is used for anti-aliasing in the output image.

[0042] Several aspects of this disclosure allow for the use of additional microgeometry information in the MSAA intermediate results. Microgeometry information can be used in conjunction with a set of pre-trained data (e.g., a set of pre-trained blend weight vectors, each pre-trained blend weight vector containing pre-trained blend weights for a particular geometric pattern) trained using high-quality reference images to obtain high-resolution outputs such as higher-resolution images. Since the costly training process is performed before the runtime loop, the complexity of runtime computations is minimized, and performance is high even on low-power devices.

[0043] Figure 1 shows a block diagram of an electronic device (100) according to some embodiments of the present disclosure. The electronic device (100) may include a graphics processing unit (GPU) (130) configured to perform (image resolution) upscaling based on MSAA intermediate results in an MSAA intermediate buffer.

[0044] The electronic device (100) may be any suitable electronic device such as a desktop computer, laptop computer, smartphone, gaming device, wearable device, portable computing device, or virtual reality headset. The electronic device (100) may run any suitable video game or similar real-time rendering application that may require a very high-performance super-resolution solution. In some examples, the electronic device (100) may have a tight budget for memory and computation, but may still be configured to run a real-time rendering application that requires super-resolution image quality.

[0045] The electronic device (100) includes a GPU (130) and a memory system (140). The memory system (140) includes memory space allocated for various storage purposes. For example, the memory system (140) includes memory (141) allocated to store a lookup table of blend weights (151), memory (142) allocated for an MSAA intermediate buffer (152), and memory (143) allocated to store an upscaled image (153) with an upscaled resolution for the final output. Memory (141), memory (142), and memory (143) may be the same type of memory or different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, hard drive, and solid state drive (SSD). In some examples, memory (141), memory (142), and memory (143) are integrated on the same chip as the GPU (130). In some examples, memory (141), memory (142), and memory (143) are implemented using off-chip memory located on a different chip from the GPU (130). In some examples, parts of memory (141), memory (142), and memory (143) are on the same chip as the GPU (130), while other parts of memory (141), memory (142), and memory (143) are implemented using off-chip memory located on a different chip from the GPU (130).

[0046] Generally, GPUs, such as GPU(130), can perform MSAA (Multi-Simulation Anti-Aliasing) during rasterization. To perform normal MSAA, the GPU may sample the signal at a spatial rate that produces more detail than the intended output image, such as the output image displayed on a screen with a screen resolution. For example, if the screen resolution is 800x600, the pixel resolution (number of pixels in 2D space) of the intended output image is represented as 800x600, and the GPU may use more samples per pixel to sample, for example, the edges of primitives. Furthermore, to perform MSAA, the GPU may store the oversampled signal in a buffer called an MSAA intermediate buffer. For example, the MSAA intermediate buffer may store microgeometric edge information. The GPU can process the oversampled signal and then, for example, use a reconstruction filter to downsample the oversampled signal to the screen resolution, and the downsampled signal can be the normal MSAA output image, which can be displayed on a screen.

[0047] According to some aspects of this disclosure, when the GPU(130) performs MSAA, memory(142) is configured as an MSAA intermediate buffer(152) that stores microgeometric edge information. For example, the MSAA intermediate buffer(152) stores geometry information for multiple samples within a pixel. The GPU(130) can access memory(142) for the microgeometric edge information in the MSAA intermediate buffer(152) and, based on the microgeometric edge information, can generate an upscaled image(153) having a higher resolution than a normal MSAA output image. The upscaled image(153) can be stored in memory(143).

[0048] Specifically, in some examples of MSAA, coverage and occlusion tests may be performed at higher resolutions than usual, such as 2x, 4x, 8x, 16x, etc., of the screen resolution. In the implementation, N sample points are generated within a pixel, where N is the multisample rate. Samples within a pixel may be called subsamples, fragments, or subpixel samples. In some examples, subsamples may be generated according to a sampling pattern. In some examples, a standard multisample pattern is defined. A standard multisample pattern may include a standard 2-sample pattern (e.g., a pattern for 2 samples within a pixel), a standard 4-sample pattern (e.g., a pattern for 4 samples within a pixel), a standard 8-sample pattern (e.g., a pattern for 8 samples within a pixel), a standard 16-sample pattern (e.g., a pattern for 16 samples within a pixel), and so on.

[0049] Figure 2 shows a diagram of a pixel (200) with a sampling pattern in several examples. The center of the pixel is indicated by (201). In the examples in Figure 2, the sampling pattern includes four sampling points (211) to (214). Pixel (200) is said to have a size of 1 × 1, representing one pixel width and one pixel height. If the center (201) has coordinates (0,0), sampling point (211) has coordinates (1 / 8, 3 / 8) in pixel units, sampling point (212) has coordinates (-3 / 8, 1 / 8) in pixel units, sampling point (213) has coordinates (3 / 8, -1 / 8) in pixel units, and sampling point (214) has coordinates (-1 / 8, -3 / 8) in pixel units. The sampling pattern in Figure 2 may be referred to as a standard 4-sample pattern, or a 4x rotating grid supersampling (RGSS) sampling pattern. In the case of a 4xRGSS sampling pattern, sampling points within a pixel do not share vertical or horizontal lines with any other sampling points within the same pixel. Sampling points in a 4xGRGS sampling pattern, such as sampling points (211) to (214), can contain nearly the same level of detail vertically and horizontally as a 4x4 grid-based sampling pattern used in rendering with FPAA. Rendering with FPAA requires 16 times the memory space of rendering without anti-aliasing. For example, using a 4xRSGS sampling pattern can produce a geometric resolution close to that of a 4x4 grid-based sampling pattern, and rendering with a 4xRSGS sampling pattern can consume four times the memory space of rendering without anti-aliasing, while reducing the number of shading costs compared to rendering with FPAA.

[0050] In some cases, MSAA is a hardware-optimized rendering mode supported by most GPU 3D graphics accelerators. In MSAA mode, within a pixel area, the GPU detects fragments (e.g., subsamples) shared by the same geometry, and the geometry within the pixel is shaded only once, rather than for each fragment covering the geometry, so that the shaded fragments within the pixel can be broadcast. In one example, when three of four subsamples within a pixel are covered (within coverage) by a first triangle and a fourth subsample within the pixel is covered by a second triangle, the centers of the three subsamples can be shaded based on the first triangle, and the shading results can be stored for each of the three subsamples. The fourth subsample can be shaded based on the second triangle. Therefore, in MSAA mode, the shading cost is lower than rendering with FPAA, which shades each subsample.

[0051] In some cases, by using an RGSS sampling pattern, the MSAA intermediate buffer can actually contain far more geometric information than a regular upscaled buffer. For example, for an 800x600 resolution output image using a 4xRPSS sampling pattern, the MSAA intermediate buffer has the same memory footprint as a regular 1600x1200 image, but the 4xRPSS sampling pattern allows the MSAA intermediate buffer to actually provide a geometric resolution close to 3200x2400.

[0052] Generally, in a typical MSAA, the geometric information of sampling points (211) to (214) is stored as an intermediate image, such as in an MSAA intermediate buffer. In a typical MSAA, sampling points (211) to (214) can be appropriately downsampled to produce a single output pixel. Several aspects of this disclosure provide techniques for using an MSAA intermediate buffer to reconstruct an output image with a higher resolution than the intended output image of a typical MSAA. For example, the 4xRSGS sampling pattern in Figure 2, which includes subsamples occupying sparse and irregular positions, can cover four rows and four columns and can provide geometric information at almost the same level of detail as a 4x4 axis-aligned 2D grid-based sampling pattern. In one example, based on the geometric information of an MSAA intermediate buffer for an intended output image of 800x600, several techniques provided by this disclosure can reconstruct higher resolution output images such as 1600x1200, 3200x2400, etc.

[0053] According to several aspects of this disclosure, the mapping of blend weights to surrounding subsample patterns is appropriately pre-trained using a data-driven approach. In the example in Figure 1, the mapping is stored as a lookup table (151) that associates a blend weight vector with a surrounding subsample pattern for each of several locations within a low-resolution pixel. The multiple locations may correspond to locations of higher-resolution output pixels. The blend weight vector corresponding to the surrounding subsample pattern contains the blend weights of each candidate subsample among the subsamples buffered in the MSAA intermediate buffer (152). The surrounding subsample pattern is formed based on the surrounding subsamples of a pixel. The surrounding subsamples are among the subsamples buffered in the MSAA intermediate buffer. The candidate subsamples may be the same subsamples as the surrounding subsamples, or they may be different from the surrounding subsamples.

[0054] In some cases, during rendering of images with a higher resolution than the output image of a normal MSAA, the GPU(130) performs MSAA, and the geometry information of subsamples of each pixel in the output image of the normal MSAA is buffered in an MSAA intermediate buffer (152). For example, the MSAA intermediate buffer (152) contains the geometry information of multiple subsamples within a pixel in the output image of the normal MSAA. Furthermore, a lookup table (151) is loaded into memory (141) accessible by the GPU(130). Furthermore, for pixels in the normal image, the GPU(130) determines a surrounding subsample pattern based on surrounding subsamples in the surrounding region of the pixel. Furthermore, according to the lookup table (151), the GPU(130) determines a blending vector associated with the surrounding subsample pattern for the position within the pixel. The blending vector contains blend weights for candidate subsamples of the pixel. The GPU(130) may calculate the value of the output pixel at that position based on a weighted sum of candidate subsamples based on the blend weights of the subsamples. The GPU (130) can store the calculated values ​​of higher-resolution output pixels, such as the upscaled image (153) in Figure 1, in memory (143).

[0055] The surrounding subsample pattern (also referred to as the search pattern in some examples) can be determined by any appropriate technique. In some examples, the surrounding subsample pattern is generated based on subsample differences and is referred to as the subsample difference pattern.

[0056] Figure 3 shows Figure (300), illustrating a technique for determining subsample difference patterns in several examples. Figure 3 shows pixel (301) and its four adjacent pixels (302) to (305) in the intended output image of rendering using normal MSAA. When a 4xRGSS sampling pattern is used, each of pixels (301) to (305) contains four subsamples (also referred to as fragments in some examples), as indicated by circles (including filled black and open circles) in Figure 3. Note that while a 4xRGSS sampling pattern is used in Figure 3, other appropriate sampling patterns may be used for MSAA. In Figure 3, small squares (including small filled black and open squares) represent locations without subsamples (also referred to as locations of missing subsamples). In some examples, the GPU(130) may reconstruct subsamples of positions represented by small filled black squares within pixel(301) based on candidate subsamples, such as those represented by filled black circles.

[0057] In some examples, a sample difference pattern is generated for each pixel, such as pixel (301), according to surrounding subsamples. Surrounding subsamples may include a subsample at pixel (301) and subsamples at neighboring pixels such as adjacent pixels (302) to (305). In the example in Figure 3, the sample difference pattern is generated based on eight subsamples (a) to (h), shown as filled black circles. In the example in Figure 3, the sample difference pattern contains 8 bits, each bit representing the result of comparing two of the eight subsamples. In one example, each bit of the 8-bit pattern is formed by comparing the color, brightness, or depth of nearby MSAA subsamples (also referred to as fragments in some examples). In Figure 3, the 8-bit pattern contains bits A to H, each corresponding to the arrows in Figure 3. Each bit X (where X is one of bits A to H) can be calculated according to equation (1).

number

[0058] For example, with respect to bit A, if the difference in brightness between the subsample (a) at the start point of the arrow and the subsample (b) at the end point is greater than the threshold, bit A is 1, and if the difference in brightness between the subsample (a) at the start point of the arrow and the subsample (b) at the end point is less than or equal to the threshold, bit A is 0.

[0059] The surrounding subsample pattern can be formed by any number of subsamples and can contain any number of bits. In some examples, the number of bits (D) in the surrounding subsample pattern is determined based on the dictionary size 2^D for fitting into memory. For example, if D is 12, the dictionary size is 4096 for each position of the missing subsample, which can correspond to the number of blend weight vectors in the lookup table. Also, any suitable comparison pattern can be used as long as the comparison pattern covers pixels such as pixel (301) reasonably well.

[0060] According to several aspects of this disclosure, upscaling based on MSAA is performed using a data-driven approach involving a training process and a runtime upscaling process. The training process is performed by any suitable device, such as an electronic device (100), a server device, or any other device, to determine a lookup table before the runtime upscaling process for rendering higher resolution images. The training process for the lookup table takes place before the runtime upscaling process and does not affect runtime resources such as memory usage and computational complexity of the runtime upscaling process; the training process is referred to as a pre-training process in some examples. After the electronic device (100) has determined the lookup table, it may store the lookup table in its file system. If the lookup table is generated by another device, it may be appropriately transmitted to the electronic device (100) and stored in its file system. The lookup table in the electronic device (100)'s file system may be loaded into memory (141) all at once for upscaling. In one example, memory (141) is configured to have a high-speed access rate for the GPU (130).

[0061] According to one aspect of this disclosure, the training process uses a high-resolution image rendered with the same camera settings as a low-resolution image output from rendering with normal MSAA, and the high-resolution image is used as a reference image. In one example, the resolution of the reference image is 4x the low-resolution image in each dimension for 4xMSAA (4 subsamples per pixel). In another example, the resolution of the reference image is 2x the low-resolution image in each dimension for 2xMSAA (2 subsamples per pixel). In some examples, the MSAA uses an RGSS sampling pattern, where each subsample occupies a unique row and column. In one example, the intended output image of normal MSAA is low-resolution, and a reference image with 4x pixels in each dimension is generated to create a lookup table for use with 4xMSAA. For example, if the low resolution is 800x600, a reference image with a resolution of 3200x2400 is used for 4xMSAA. The reference image can be generated by any appropriate technique, such as rendering at a high resolution such as 3200x2400.

[0062] During the training process, low-resolution images are rendered using MSAA. Each pixel in the low-resolution image contains a position corresponding to a pixel in the reference image. Some positions have available subsamples in the MSAA intermediate buffer, while some positions have missing subsamples that are not available in the MSAA intermediate buffer. In some examples, for each position of a missing subsample (also called a missing fragment) at the current pixel (301) (represented by small filled black squares in Figure 3), the best-matching subsample is determined from the available MSAA subsamples (also called MSAA fragments). The available MSAA subsamples are referred to as candidate subsamples in some examples. In some examples, the candidate subsamples include subsamples from the current pixel. For example, in the case of 4xMSAA, the candidate subsamples could be four subsamples such as subsamples (b), (d), (f), and (h) at pixel (301). In some examples, the candidate subsamples include the same set of subsamples used to construct the subsample difference pattern, and therefore, no new subsamples need to be loaded. For example, in the example in Figure 3, the candidate subsamples could be the eight subsamples (a) to (h) shown by the filled black circles.

[0063] Specifically, during the training process, for each location of a missing subsample, one of the candidate subsamples is determined to be the candidate that best matches the corresponding pixel in the reference image for that location. In one example, the best-matching candidate has the smallest error compared to the corresponding pixel in the reference image. Then, votes for the best-matching candidate are collected and associated with a subsample difference pattern.

[0064] Figure 4 shows Figure (400) illustrating the votes for the most matching candidate in one example. The subsample difference pattern is determined according to subsamples (a) to (h) as shown in Figure 3, and the subsample difference pattern contains 8 bits. When a 4xRGSS pattern is used in MSAA, each current pixel contains 12 positions of missing subsamples numbered (1) to (12). Candidate subsamples contain the same set of subsamples (a) to (h) and form a subsample difference pattern. Thus, for each position of a missing subsample, there are 256 subsample difference patterns, and for each subsample difference pattern, 8 candidate subsamples may be voted. The total number of votes for each candidate subsample can be collected for each position of the missing subsample in each subsample difference pattern. In one example, for the location of a missing subsample in pixel (301) of a low-resolution image (for example, shown as 1 in Figure 3), a different pattern of subsamples for pixel (301) is determined to be "00000000". Then, candidate subsample (a) is determined to be the candidate that best matches the corresponding pixel in the reference image. Then, votes are added to the vote count for candidate subsample (a) in Figure 4, and for example, the total vote count for candidate subsample (a) may increase from 1 to 2 in Figure 4.

[0065] In one example, if 8 bits are used as the subsample difference pattern, there are 256 subsample difference patterns for the current pixel. When a 4xRGSS pattern is used, if there are 12 locations with missing subsamples and the number of candidate samples is 8, the storage of entries for collecting votes could be 256 × 12 × 8. In some examples, when N bits are used for different patterns of subsamples, M is the number of MSAA samples, and C is the number of candidate samples, the maximum storage of entries for collecting votes can be calculated as (2^N) × M × M × C. At each location of a missing subsample, the best candidate subsample is found, and a vote for one of the candidate subsamples can be added to the corresponding entry. In some examples, the final result of the collected votes can be normalized to obtain a C-length vector representing the blended weights of the candidate subsamples.

[0066] Note that the storage requirements for collecting votes are resolution-independent. In some examples, the results of the training process include a C-length vector associated with each position of the missing subsample in each subsample difference pattern. Note that the available subsamples in MSAA may have a default C-length vector with full weights for the subsample itself.

[0067] Furthermore, while the training process in the above example is performed for locations with missing subsamples, it should be noted that the training process can be performed for all locations, including locations with missing subsamples and locations with available subsamples from the MSAA.

[0068] In some cases, the results of the training process can be appropriately downsampled. For example, the training results support upscaling to M times the resolution, where M is the number of MSAA subsamples. For instance, a 960x540 rendering with 4xMSAA can be upscaled to a resolution of 3840x2160 based on the training results. However, in some cases, such high-resolution output is not always desired. In one example, blend weights, such as C-length vectors, can be downsampled.

[0069] Figure 5 shows diagrams of blend weight downsampling in several examples. In Figure 5, the low-resolution pixels (500) contain four subsamples of the 4xRGSS pattern from the MSAA operation. The four subsamples are indicated by circles. The pixels (500) contain 12 locations with missing subsamples, indicated by small squares. The training process may generate a C-length vector as the blend weight for the missing subsamples. The C-length vector can be used for 4x4 upscaling. In some examples, 2x2 upscaling is preferable. In some examples, the C-length vector can be downsampled by 2x2 to generate a downsampled C-length vector.

[0070] For example, to perform a 2x2 upscaling, four output pixels need to be generated at pixel (500), and the positions of the four output pixels are indicated by triangles. In some examples, the C-length vector associated with an output pixel is calculated by blending nearby C-vectors. For example, the C-length vector for output pixel (501) may be calculated as the average of the C-length vectors of subsamples (502), (503), (504), and (505). Note that the C-length vector for output pixel (501) may be calculated by other suitable algorithms, such as more sophisticated algorithms. In the example in Figure 5, the downsampling result includes the C-length vector for each of the four output pixels (indicated by triangles) in each of the subsample difference patterns. The result of the downsampled blend weights can be used to perform a 2x2 upscaling. For example, using downsampled blend weights, a 960x540 resolution input can produce a 1920x1080 resolution in 4xMSAA.

[0071] In some cases, the training process converges after matching with multiple reference images. In some cases, the C-length vector can be compressed and then stored as data to be loaded for runtime use. In some cases, the C-length vector can be downsampled, compressed, and then stored as data to be loaded for runtime use.

[0072] In some examples, C-length vectors can be stored in the form of a lookup table. For example, C-length vectors can be indexed according to subsample difference patterns and subsample positions. The lookup table contains C-length vectors indexed according to subsample difference patterns and subsample positions. For example, the index is determined according to the subsample difference pattern and subsample positions, and the C-length vectors in the index can be output as blend weights associated with the subsample difference pattern and subsample positions.

[0073] In some examples, C-length vectors can be organized into multiple lookup tables, each associated with a subsample location. Each lookup table associated with a subsample location contains C-length vectors associated with a subsample difference pattern. Each lookup table may be 2^N in size, containing 2^N C-length vectors associated with a subsample difference pattern, where N is the number of bits in each subsample difference pattern. The index of a C-length vector can be determined based on the subsample difference pattern.

[0074] According to one aspect of this disclosure, during a runtime upscaling process, one or more lookup tables are loaded and a low-resolution image is rendered using MSAA. Multiple locations at each pixel of the low-resolution image may correspond to pixels in the upscaled image to be output as a result of the runtime upscaling process. Some locations have available subsamples, for example, in the MSAA intermediate buffer, while some locations with missing subsamples are not available by rendering the low-resolution image using MSAA. In some examples, for the current pixel (e.g., pixel (301)), a subsample difference pattern is constructed according to the available subsamples in the surrounding region in the MSAA intermediate buffer (e.g., surrounding region (311) in Figure 3). An index is then determined according to the subsample difference pattern. The lookup table associated with the location with the missing subsample is retrieved according to the index to determine a C-length vector. The C-length vector contains blend weights for candidate subsamples in the MSAA intermediate buffer. The missing subsample at the location is then reconstructed according to the blend weights and candidate subsamples in the MSAA intermediate buffer.

[0075] Figure 6 shows a diagram illustrating runtime upscaling in several examples. Figure 6 shows low-resolution pixels (600), which in some examples are also referred to as upscaling work units (600). In Figure 6, the upscaling work unit (600) contains 2x2 output pixels for the upscaled image. The 2x2 output pixels are represented by triangles in Figure 6. In some examples, each output pixel has a unique (2^N)-sized lookup table associated with the output pixel's position, where N is the number of bits in the subsample difference pattern. Each entry in the lookup table is a C-sized vector, where C is the number of candidate subsamples. Note that the candidate samples are not limited to the current upscaling work unit, and often some of the candidate subsamples are fetched outside of the current upscaling work unit.

[0076] In some examples, for the current upscaling work unit (600), N surrounding subsamples are fetched into an array Subsamples[N], where N is a predetermined number of bits. Based on the N surrounding subsamples, a subsample difference pattern (e.g., N bits) is calculated. The subsample difference pattern corresponds to an index in a lookup table associated with the output pixel position.

[0077] In some examples, for the current upscaling work unit (600), C candidate subsamples are fetched into array Subsamples[C] for blending calculations. In one example, the candidate subsamples may be the same as the surrounding subsamples used to determine the subsample difference pattern, and the fetching of C candidate subsamples may be skipped. In some examples, N surrounding subsamples may include all of the C candidate subsamples, and unused subsamples may simply be ignored. In some examples, C candidate subsamples may include additional subsamples to the N surrounding subsamples, and therefore additional subsamples are fetched.

[0078] Furthermore, in some examples, for each output pixel in the upscaling work unit (600), a lookup table associated with the output pixel is placed. The lookup table is searched according to the subsample difference pattern to find a C-sized vector associated with the subsample difference pattern in the lookup table. The C-sized vector represents the respective blend weight for the candidate subsamples. Subsamples[C] are blended according to the blend weights indicated by the C-sized vector. The blend result is for the output pixels of the upscaled image. In one example, the blend result is stored in memory (143) in the memory space allocated to the output pixels of the upscaled image.

[0079] According to one aspect of this disclosure, the runtime upscaling process uses low-resolution rendering with MSAA, lookup operations, multiplication, and addition operations, and the runtime upscaling process can be performed quickly at runtime by most GPUs.

[0080] According to one aspect of this disclosure, in rendering using normal MSAA, alpha testing is performed once by default for covered subsamples within a pixel, resulting in only one alpha output for all subsamples. The alpha test results do not have geometric edges, and normal rendering using MSAA does not, of course, anti-alias the alpha test object. According to one aspect of this disclosure, a technique called alpha-to-coverage may be used. Alpha-to-coverage can replace alpha blending values ​​with a coverage mask. For example, when multisampling is used, alpha-to-coverage may generate bits for subsamples within a pixel to indicate the transparency of the subsamples.

[0081] According to several aspects of this disclosure, MSAA-based upscaling techniques can efficiently acquire additional geometric edge information for upscaling at virtually no additional cost by utilizing MSAA, which is supported on most GPUs. MSAA-based upscaling techniques can use high-quality images to train blend weights offline and use efficient computations such as comparison, lookup, and blending for upscaling at runtime. MSAA-based upscaling techniques can support up to NxN resolution upscaling factors when using NxMSAA. MSAA-based upscaling techniques can support proper upscaling of alpha test renderings. Furthermore, MSAA-based upscaling does not require multi-frame input images (multi-image super-resolution, MISR), and therefore can significantly reduce implementation and computational complexity, enabling efficient rendering even on low-power mobile devices.

[0082] Figure 7 shows a flowchart outlining process (700) according to one embodiment of the present disclosure. Process (700) is a training process. In some embodiments, process (700) is implemented as a software instruction, and therefore, when a processing circuit (e.g., a GPU) executes a software instruction, the processing circuit performs process (700). The process starts at (S701) and proceeds to (S710).

[0083] In (S710), rendering is performed using MSAA to generate a first image of a first resolution, and the geometry information of subsamples of the MSAA is buffered in an MSAA intermediate buffer, which contains the geometry information of multiple subsamples of pixels of the first resolution.

[0084] In (S720), a reference image for the first image is generated. The reference image has the same camera settings as the first image and a second resolution that is higher than the first resolution.

[0085] In (S730), a vote for candidate subsamples for each output pixel position at the first resolution is collected for each of the surrounding subsample patterns based on the reference image and the MSAA intermediate buffer.

[0086] In some examples, for a first pixel of a first image, a first surrounding subsample pattern is determined based on a first subsample from subsamples buffered in the MSAA intermediate buffer, where the first subsample lies in the surrounding region of the first pixel. If a position within the first pixel has a corresponding pixel in the reference image, the position is referred to as the output pixel position. For the output pixel position, a candidate subsample is selected from the first candidate subsamples, where the first candidate subsample is from subsamples buffered in the MSAA intermediate buffer. The selected candidate subsample has the smallest error with respect to the corresponding pixel in the reference image. Votes are then added to the first surrounding subsample pattern and the determined candidate subsample associated with the output pixel position.

[0087] In some examples, votes are collected based on multiple images at a first resolution and their baseline images at a second resolution that is lower than the first resolution.

[0088] In (S740), the blend weight for each of the surrounding subsample patterns is determined for each output pixel position based on the collected votes. In some examples, the votes for the surrounding subsample patterns and candidate subsamples associated with the output pixel position are normalized to form a blend weight vector of blend weights.

[0089] In some examples, blend weights may be placed in a lookup table associated with the output pixel locations and used to upscale to a second resolution. In some examples, blend weights may be downsampled for downsampled output pixel locations, and the downsampled blend weights may be used to upscale to a resolution lower than the second resolution but higher than the first resolution.

[0090] Next, the process proceeds to (S799) and terminates.

[0091] Process (700) may be appropriately adapted. The steps of Process (700) may be changed and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0092] Figure 8 shows a flowchart outlining process (800) according to one embodiment of the present disclosure. Process (800) is a runtime upscaling process. In some embodiments, process (800) is implemented as a software instruction, and therefore, when a processing circuit (e.g., a GPU) executes a software instruction, the processing circuit performs process (800). The process starts at (S801) and proceeds to (S810).

[0093] In (S810), rendering is performed using MSAA to generate a first image of a first resolution. The geometry information of the MSAA subsamples is buffered in an MSAA intermediate buffer, and the MSAA intermediate buffer contains geometry information of multiple subsamples for each pixel in the first image of the first resolution.

[0094] In (S820), for the first pixel of the first image, the first surrounding subsample pattern is determined based on the first subsample among the subsamples buffered in the MSAA intermediate buffer, and the first subsample lies within the surrounding region of the first pixel.

[0095] In (S830), a lookup table of blend weight vectors for at least a first position is accessed and loaded into memory, for example, for GPU access. The lookup table associates the blend weight vectors with the surrounding subsample pattern for the first position. In some examples, the blend weight vectors associated with the surrounding subsample pattern are pre-trained. The first position is one of several positions that define the output pixel positions in the pixels of the first image for upscaling the pixels to multiple output pixels in an upscaled image with a higher resolution than the first image.

[0096] In (S840), a first blend weight vector corresponding to the first surrounding subsample pattern is determined according to the lookup table. The first blend weight vector includes the first blend weight for each of the first candidate subsamples in the MSAA intermediate buffer.

[0097] In (S850), an upscaled image corresponding to the first image is generated, and the value for the first output pixel of the upscaled image generated from at least the first position of the first pixel of the first image is calculated as a weighted sum of the first candidate subsamples according to the first blend weights.

[0098] In some examples, the multiple subsamples generated by rendering with MSAA for a pixel at a first resolution have a Rotated Grid Supersampling (RGSS) pattern. In one example, rendering with MSAA generates four subsamples of a 4xRGSS pattern for a pixel at a first resolution.

[0099] In some examples, the first subsample in the region surrounding the first pixel includes multiple subsamples in the first pixel. In some examples, the first subsample in the region surrounding the first pixel also includes one or more subsamples in one or more neighboring pixels of the first pixel.

[0100] In some examples, the first surrounding subsample pattern is a subsample difference pattern. In one example, the difference between two subsamples in the first subsample is calculated, and the bits in the first surrounding subsample pattern are determined based on the difference. In some examples, the difference can be a chrominance difference, a luminance difference, or a depth difference.

[0101] In some examples, the first subsample used to generate the first surrounding subsample pattern is from the same set of subsamples as the first candidate subsample.

[0102] In some examples, the first subsample used to generate the first surrounding subsample pattern includes the first candidate subsample.

[0103] In some examples, the first candidate subsample includes the first subsample used to generate the first surrounding subsample pattern.

[0104] In some examples, the lookup table contains a set of blend weight vectors for at least a first position, and the index of the blend weight vector in the lookup table is determined by the surrounding subsample pattern associated with the blend weight vector. The set of blend weight vectors for the first position can be pre-trained.

[0105] Next, the process proceeds to (S899) and terminates.

[0106] Process (800) may be appropriately adapted. The steps of Process (800) may be changed and / or omitted. Additional steps may be added. Any appropriate implementation order may be used.

[0107] Figure 9 shows a flowchart outlining process (900) according to one embodiment of the present disclosure. Process (900) is a runtime upscaling process. In some embodiments, process (900) is implemented as a software instruction, and therefore, when a processing circuit (e.g., a GPU) executes a software instruction, the processing circuit performs process (900). The process starts at (S901) and proceeds to (S910).

[0108] In (S910), a first multi-sample anti-aliasing (MSAA) intermediate buffer is obtained from rendering the first image, and the first MSAA intermediate buffer contains multiple subsamples for each pixel of the first image.

[0109] In (S920), for the first pixel of the first image, the first surrounding subsample pattern is determined based on the first subsample in the surrounding region of the first pixel.

[0110] In (S930), a first blend weight vector for a first position is obtained based on a first surrounding subsample pattern. The first position is one of several positions that define the output pixel positions within the first pixel for upscaling the first pixel to multiple output pixels in an upscaled image with a higher resolution than the first image. The first blend weight vector includes the first blend weight for each of the first candidate subsamples associated with the first pixel.

[0111] In (S940), an upscaled image corresponding to the first image is generated. The value of the first output pixel of the upscaled image, generated from the first position in the first pixel of the first image, is calculated as a weighted sum of the first candidate subsamples according to the first blend weights.

[0112] In some examples, rendering a first image generates multiple subsamples for each pixel of the first image, each with a Rotated Grid Supersampling (RGSS) pattern. In one example, rendering generates four subsamples for each pixel of the first image, each with a 4xRGSS pattern.

[0113] In some examples, the first subsample in the region surrounding the first pixel includes multiple subsamples in the first pixel and one or more subsamples in one or more neighboring pixels of the first pixel.

[0114] In some examples, the first ambient subsample pattern is a subsample difference pattern. To determine the first ambient subsample pattern, in one example, the difference between two subsamples in the first subsample is determined, and the bits in the first ambient subsample pattern are determined based on this difference. The difference is at least one of chrominance, luminance, and depth difference.

[0115] In some examples, the first subsample used to generate the first surrounding subsample pattern is from the same set of subsamples as the first candidate subsample.

[0116] In some examples, the first subsample used to generate the first surrounding subsample pattern includes the first candidate subsample.

[0117] In some examples, the first candidate subsample includes the first subsample used to generate the first surrounding subsample pattern.

[0118] In some examples, to obtain the first blend weight vector, the index to the lookup table is determined according to the first surrounding subsample pattern, and the lookup table contains a set of blend weight vectors indexed according to the surrounding subsample pattern.

[0119] Next, the process proceeds to (S999) and terminates.

[0120] Process (900) may be appropriately adapted. The steps of Process (900) may be changed and / or omitted. Additional steps may be added. Any appropriate implementation order may be used.

[0121] The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, Figure 10 shows a computer system (1000) suitable for implementing a particular embodiment of the disclosed subject matter.

[0122] Computer software can be coded using any suitable machine code or computer language that can follow mechanisms such as assembly, compilation, and linking to create code that includes instructions that can be executed directly or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0123] Instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0124] The components shown in Figure 10 for the computer system (1000) are essentially illustrative and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependencies or requirements relating to any one or combination of components shown in the exemplary embodiments of the computer system (1000).

[0125] The computer system (1000) may include a human interface input device. Such a human interface input device may respond to input from one or more human users, such as tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voices, applause, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographic images taken from a still image camera, etc.), and video (2D video, 3D video including stereoscopic video, etc.).

[0126] The input human interface device may include one or more of the following (only one of each is shown): a keyboard (1001), a mouse (1002), a trackpad (1003), a touchscreen (1010), a data glove (not shown), a joystick (1005), a microphone (1006), a scanner (1007), and a camera (1008).

[0127] The computer system (1000) may include a human interface output device. Such a human interface output device may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such a human interface output device may include tactile output devices (e.g., tactile feedback via a touchscreen (1010), data glove (not shown), or joystick (1005), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (1009), headphones (not shown)), visual output devices (e.g., screens (1010) including CRT screens, LCD screens, plasma screens, OLED screens), each with or without touchscreen input functionality, each with or without tactile feedback functionality, some of which may be capable of outputting two-dimensional visual output or more than three-dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0128] The computer system (1000) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1020) having CD / DVD or similar media (1021), thumb drives (1022), removable hard drives or solid-state drives (1023), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0129] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter of this disclosure does not include a transmission medium, carrier wave, or other transient signal.

[0130] The computer system (1000) may also include an interface (1054) to one or more communication networks (1055). The networks may be, for example, wireless, wired, or optical. The networks may further include local, wide-area, metropolitan, vehicle and industrial, real-time, and latency-tolerant. Examples of networks include local area networks such as Ethernet® and Wi-Fi; cellular networks such as GSM®, 3G, 4G, 5G, and LTE; wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicle and industrial networks including CANBus. Some networks generally require an external network interface adapter attached to a general-purpose data port or peripheral bus (1049) (e.g., a USB port on the computer system (1000)), while others are generally integrated into the core of the computer system (1000) by being attached to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1000) may communicate with other entities. Such communication may be one-way reception only (e.g., broadcast television), one-way transmission only (e.g., from a CANbus to a CANbus device), or two-way communication, such as to other computer systems using local or wide-area digital networks. Certain protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0131] The aforementioned human interface devices, human-accessible memory devices, and network interfaces may be mounted on the core (1040) of the computer system (1000).

[0132] The core (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (1043), hardware accelerators for specific tasks (1044), graphics adapters (1050), and the like. These devices, along with read-only memory (ROM) (1045), random access memory (1046), and internal mass storage devices such as hard drives and SSDs (1047) that are not accessible to the internal user, may be connected via a system bus (1048). In some computer systems, the system bus (1048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (1048) or via a peripheral bus (1049). For example, a screen (1010) may be connected to a graphics adapter (1050). The peripheral bus architecture includes PCI, USB, and others.

[0133] The CPU (1041), GPU (1042), FPGA (1043), and accelerator (1044) can execute certain instructions that, in combination, may form the aforementioned computer code. This computer code may be stored in ROM (1045) or RAM (1046). Temporary data may also be stored in RAM (1046), while persistent data may be stored, for example, in internal mass storage (1047). By using cache memory that can be closely associated with one or more CPUs (1041), GPUs (1042), mass storage (1047), ROM (1045), RAM (1046), etc., high-speed storage and retrieval to any of the memory devices may be enabled.

[0134] Computer-readable media may contain computer code for performing various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0135] For example, but not limited to, a computer system having architecture (1000), specifically a core (1040), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as described above, as well as media associated with specific storage devices of the core (1040) of a non-transient nature, such as core internal mass storage (1047) or ROM (1045). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1040). The computer-readable media may include one or more memory devices or chips according to specific needs. The software may cause the core (1040), specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1046) and modifying such data structures according to processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logic hardwired to or otherwise embodied in a circuit (e.g., an accelerator (1044)), which may, in place of or in conjunction with software, execute a particular process or a particular part of a particular process described herein. References to software may, where appropriate, encompass logic, and vice versa. References to computer-readable media may, where appropriate, encompass a circuit (such as an integrated circuit (IC)) that stores software for execution, a circuit that embodies logic for execution, or both. This disclosure encompasses any appropriate combination of hardware and software.

[0136] While this disclosure has described several exemplary embodiments, there are many modifications, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will therefore understand that numerous systems and methods not expressly shown or described herein can be devised to embody the principles of this disclosure and thus fall within its spirit and scope.

Claims

1. A method for image rendering performed by a graphics processing unit (GPU) in an electronic device, A step of obtaining a first multi-sample anti-aliasing (MSAA) intermediate buffer from rendering a first image, wherein the first MSAA intermediate buffer includes a plurality of subsamples for each pixel of the first image. The steps include determining a first surrounding subsample pattern for a first pixel of the first image based on a first subsample in the surrounding region of the first pixel, Steps of obtaining a first blend weight vector for a first position within a first pixel based on the first surrounding subsample pattern, wherein the first position is one of a plurality of positions that define output pixel positions within the first pixel for upscaling the first pixel to a plurality of output pixels in an upscaled image with a higher resolution than the first image, and the first blend weight vector comprises a first blend weight for each of the first candidate subsamples associated with the first pixel. A step of generating an upscaled image corresponding to the first image, wherein the value for a first output pixel of the upscaled image at a first position in the first pixel of the first image is calculated as a weighted sum of the first candidate subsamples according to the first blend weights, A method that includes this.

2. The method according to claim 1, wherein the rendering of the first image generates a plurality of subsamples having a rotation grid supersampling (RGSS) pattern for the pixels of the first image.

3. The method according to claim 2, wherein the rendering of the first image generates four subsamples of a 4x RGSS pattern for the pixels of the first image.

4. The method according to claim 1, wherein the first subsample in the surrounding region of the first pixel includes a plurality of subsamples in the first pixel.

5. The method according to claim 4, wherein the first subsample in the surrounding region of the first pixel includes one or more subsamples in one or more adjacent pixels of the first pixel.

6. The first surrounding subsample pattern is a subsample difference pattern, and the step of determining the first surrounding subsample pattern is: The steps include determining the difference between two subsamples within the first subsample, The method according to claim 1, further comprising the step of determining a bit in the first surrounding subsample pattern based on the difference.

7. The method according to claim 6, wherein the difference is at least one of color difference, brightness difference, and depth difference.

8. The method according to claim 1, wherein the first subsample used to generate the first ambient subsample pattern is the same set of subsamples as the first candidate subsample.

9. The method according to claim 1, wherein the first subsample used to generate the first ambient subsample pattern includes the first candidate subsample.

10. The method according to claim 1, wherein the first candidate subsample includes the first subsample used to generate the first surrounding subsample pattern.

11. The step of obtaining the first blend weight vector is: The method according to claim 1, further comprising the step of determining an index to a lookup table according to the first surrounding subsample pattern, wherein the lookup table includes a set of blend weight vectors indexed according to the surrounding subsample pattern.

12. The method according to claim 11, further comprising the step of pre-training the set of blend weight vectors for the surrounding subsample pattern by using at least one reference image having a resolution greater than or equal to the upscaled image.

13. The step of pre-training the aforementioned set of blended weight vectors is: A step of obtaining a second MSAA intermediate buffer from rendering a second image having the same camera settings as the reference image, wherein the second image has the same resolution as the first image. A step of collecting votes for candidate subsamples for the surrounding subsample pattern based on the second MSAA intermediate buffer and the reference image, A step of determining the blend weights in the blend weight vector associated with the surrounding subsample pattern according to the vote of the candidate subsamples, The method according to claim 12, including the method described in claim 12.

14. The step of collecting the votes for the candidate subsample is: Regarding the position of a pixel in the second image having a corresponding pixel in the reference image, The steps include determining a first candidate subsample from the candidate subsamples that has the smallest error with respect to the corresponding pixel in the reference image, The process includes the step of adding votes to the first candidate subsample, The method according to claim 13.

15. An electronic device comprising a processing circuit configured to perform the method described in any one of claims 1 to 14.

16. A computer program for causing a computer to perform the method described in any one of claims 1 to 14.

Images