Methods, apparatus, and computer programs for image rendering

Abstract

A method for geometry buffer compression includes the step of generating compressed geometry information for buffering in a geometry buffer (G-buffer) based on the geometry information of pixels in a frame. The compressed geometry information includes a first texture component for a first group of pixels, and omits the first texture component for at least one second pixel. The second pixel has one or more neighboring pixels in the first group of pixels. Furthermore, the method includes the steps of accessing the geometry buffer to obtain the compressed geometry information and generating reconstructed geometry information for pixels based on the compressed geometry information. The reconstructed geometry information includes a first texture component for the second pixel reconstructed based on the first texture components of one or more neighboring pixels in the first group of pixels. The method then includes the step of rendering to generate an image according to the reconstructed geometry information of the pixels.

Description

[[Technical Field]]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 18 / 234,781 entitled "G-BUFFER COMPRESSION", filed on August 16, 2023, which is incorporated herein by reference in its entirety.

[0002] The present disclosure generally relates to graphics processing, and more particularly describes embodiments relating to compression and decompression techniques for geometry information in graphics processing. [[Background Art]]

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent described in this Background section, the research of the current inventor, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor implicitly admitted as prior art against the present disclosure.

[0004] In graphics processing, rendering may refer to the process of generating an image from a three-dimensional (3D) model or a two-dimensional (2D) model. In some examples, a computing system may include a graphics processing unit (GPU) to accelerate rendering, and the rendering process in a computing system may be referred to as a graphics pipeline. The graphics pipeline may include various operations such as vertex shading operations, fragment shading operations, and the like. [[Summary of the Invention]] [[Means for Solving the Problem]]

[0005] Aspects of this disclosure provide methods and apparatus for geometry buffer (G-buffer) compression (e.g., compression and decompression of geometry information for buffering in a G-buffer). In some examples, a method for geometry buffer compression includes the step of generating compressed geometry information for buffering in a geometry buffer (G-buffer) based on the geometry information of pixels in a frame. The compressed geometry information includes a first texture component for a first group of pixels, and omits the first texture component for at least one second pixel. The second pixel has one or more adjacent pixels in the first group of pixels. The compressed geometry information is buffered in a geometry buffer. Furthermore, the method includes the steps of accessing the geometry buffer to obtain the compressed geometry information and generating reconstructed geometry information for pixels based on the compressed geometry information. The reconstructed geometry information includes a first texture component for a second pixel reconstructed based on the first texture components of one or more adjacent pixels in the first group of pixels. The method then includes the step of rendering to generate an image according to the reconstructed geometry information of the pixels.

[0006] In some embodiments, the compressed geometry information includes a first texture component for a first group of pixels and a second texture component for a second group of pixels, wherein the first group of pixels has a first checkerboard pattern and the second group of pixels has a second checkerboard pattern complementary to the first checkerboard pattern. The method then includes buffering the first texture component for the first group of pixels and the second texture component for the second group of pixels in channels of a geometry buffer.

[0007] In some embodiments, compression and decompression are performed by a quad. For a 2x2 pixel quad including two first pixels in a first diagonal direction and two second pixels in a second diagonal direction, the method includes the step of buffering a first texture component of the two first pixels and a second texture component of the two second pixels into channel space for the quad. Furthermore, to generate reconstructed geometry information, the method includes the steps of determining a second texture component of the two first pixels based on a second texture component of the two second pixels, and determining a first texture component of the two second pixels based on a first texture component of the two first pixels.

[0008] In some embodiments, the method includes the steps of defining a fullscreen primitive as a triangle covering the screen space for displaying an image, and generating geometric information reconstructed by quads starting with even pixel coordinates.

[0009] In some embodiments, shader costs can be amortized by co-computation. For example, to perform rendering, the method includes the steps of: performing a first operation on a first texture component for one pixel in a first pair of pixels by a first processing unit to produce a first intermediate result; performing a second operation on a second texture component for one pixel in a second pair of pixels by a second processing unit to produce a second intermediate result; exchanging the first and second intermediate results between the first and second processing units; rendering the pixels in the first pair of pixels by the first processing unit based on the first and second intermediate results; and rendering the pixels in the second pair of pixels by the second processing unit based on the first and second intermediate results.

[0010] In some examples, the second pixel has a first adjacent pixel along the first direction and a second adjacent pixel along the second direction, and in order to generate reconstructed geometry information, the method includes the steps of determining a direction in which the function has a smaller gradient from the first and second directions, and reconstructing a first texture component for the second pixel based on one of the first and second adjacent pixels along the direction. In some examples, in order to reconstruct the first texture component for the second pixel, the method includes the step of performing an inverse math calculation of the inter-pixel gradient to obtain the first texture component of the second pixel.

[0011] In some examples, the function is predefined so that it can indicate discontinuities in surface properties. Surface properties include at least one of depth properties and material properties.

[0012] In some examples, the first texture component can be at least one of the color component, surface normal, and surface depth.

[0013] Aspects of the present disclosure also provide a non-temporary computer-readable medium that, when executed by a computer, stores instructions causing the computer to perform a method for G-buffer compression (e.g., compressing and decompressing geometric information for buffering in a G-buffer).

[0014] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description and accompanying drawings below. [Brief explanation of the drawing]

[0015] [Figure 1] This is a block diagram of a buffering module according to some embodiments of the present disclosure. [Figure 2] This is a diagram illustrating color compression in several examples. [Figure 3]This is a diagram for setting up screen primitives in several examples. [Figure 4] This is a diagram illustrating color decompression in several examples. [Figure 5] This figure shows an image for comparing the results of geometry buffer compression and related compression examples in this disclosure. [Figure 6] This is a diagram of a graphics processing pipeline according to some embodiments of the present disclosure. [Figure 7] This diagram illustrates the shader's behavior for two adjacent pixels in a gather-compute example. [Figure 8] This diagram illustrates the shader's behavior for two adjacent pixels in a compute-gather example. [Figure 9] This is a flowchart outlining the process according to several embodiments of the present disclosure. [Figure 10] This is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0016] The detailed descriptions provided below in relation to the attached drawings are intended to describe various configurations and are not intended to represent only the configurations in which the concepts described herein may be implemented. The detailed descriptions include specific details to ensure a full understanding of the various concepts. However, these concepts may be implemented without these specific details.

[0017] In this disclosure, some technical terms are used with the following definitions.

[0018] A G-buffer, also known as a geometry buffer, refers to a set of textures where each pixel represents all kinds of geometric information from a mesh object, such as albedo color, specular color, surface normal, and surface depth, especially those needed for subsequent lighting calculations.

[0019] Tile-based rendering refers to a graphics processing unit (GPU) architecture used for 3D graphics acceleration, for example, in mobile devices. Tile-based rendering divides the screen into small fragments (tiles) and may complete fragment shading for each tile before writing the tile to memory.

[0020] Vulkan refers to an open standard that is a low-overhead, cross-platform application programming interface (API) for 3D graphics and computing.

[0021] In the Vulkan concept, a render pass refers to subsequent rendering operations that occur on a predefined set of render targets and depth buffers. Each render pass may include one or more subpasses.

[0022] In the Vulkan concept, a subpass refers to a subset of a render pass, represents an individual phase of rendering within the render pass, and defines a set of input attachments, output attachments, and optional intermediate attachments. During a subpass, each pixel can fetch the output rendering data of a previous subpass as an input attachment, but only the current pixel coordinates are permitted. In other words, pixel data cannot be fetched from arbitrary coordinates. Subpasses are designed for tile-based rendering architectures.

[0023] API stands for application programming interface. Graphics APIs enable applications such as games and simulation software to efficiently access the functions of graphics acceleration hardware across vendor boundaries.

[0024] According to some aspects of this disclosure, a rendering process can store intermediate information about the fragments or pixels to be displayed. In some examples, the rendering process is performed based on deferred shading. Deferred shading refers to a method of decoupling the shading of fragments (or pixels) from the calculation of complex geometry and lighting conditions. For example, the rendering process includes a first pass and subsequent passes such as a second pass. In the first pass of the rendering process, data necessary for shading calculations is collected. The data (such as texture data) includes the position, normals, and material of each surface and is rendered into a geometry buffer (G-buffer) using render-to-texture. Render-to-texture is the process of rasterizing into a texture map for further use as a resource in subsequent render passes. The G-buffer is a collection of per-pixel textures, where each pixel contains various geometric information from, for example, a mesh object, such as albedo color, specular color, surface normals, and surface depth, which may be necessary for subsequent lighting calculations. In some examples, the G-buffer can have the same geometry as the final image (e.g., the final image produced by the rendering process), and the final image and the G-buffer (e.g., texture information in a data structure organized on a pixel-by-pixel basis) can correspond one-to-one with each other. In some examples, in subsequent passes, such as a second pass, the shader can use information from the texture information in screen space, such as the information in the G-buffer, to calculate direct and indirect lighting at each pixel.

[0025] In some examples, the G-buffer can be organized into G-buffer channels corresponding to texture components. These G-buffer channels have the same geometry as the final image and contain the texture component values ​​for pixels in the final image. For example, a G-buffer channel corresponding to surface normals contains the surface normal values ​​for pixels in the final image.

[0026] Delayed shading-based rendering processes can be employed by various applications, such as video games, to achieve photorealistic rendering results due to their decoupling capabilities. However, because the G-buffer contains multiple textures to describe all the geometric properties of a given pixel, delayed shading-based rendering processes can require relatively large memory bandwidth as the size of the G-buffer continues to increase. As the complexity of shading and lighting and the demand for photorealism increase, the G-buffer needs to store more and more data over time. According to one aspect of this disclosure, applications such as video games run on power-limited electronic devices, such as those run on GPUs in mobile phones and tablet devices. In some examples, a large G-buffer can cause the GPU to use a lot of power just to move pixel data around an external bus to and from DRAM, reducing parallelism due to limited on-chip memory, and potentially forcing a reduction in tile size in tile-based rendering.

[0027] Several aspects of this disclosure provide compression techniques for compressing G-buffers and reducing their size. In some examples, the G-buffer size can be reduced by a ratio of up to 2:1. The compression techniques can significantly reduce the storage and bandwidth requirements of the G-buffer without compromising the quality of data quantization. In some examples, the compression techniques can overcome the hardware limitations of tile-based rendering architectures, which restrict access only to the current pixel and lack the ability to explore data correlations between pixels.

[0028] In the first related example, the related G-buffer compression technique utilizes bit packing techniques to reduce data size, for example, by reducing 8 bits per channel to 6 bits per channel. The first related example may introduce some degree of quantization / banding visual artifacts, and the algorithm in the first related example is cumbersome. Also, the algorithm in the first related example operates on a single pixel and does not utilize data correlation between pixels.

[0029] In the second related example, the relevant G-buffer compression technique compresses the surface normal from 2x16 bits to 3x8 bits, a 25% reduction. However, the algorithm in the second related example is specifically designed to compress normal vectors and is not suitable for other geometric data such as color data.

[0030] According to one aspect of this disclosure, adjacent pixels are likely to have very similar data because they may share the same material or geometry. This disclosure can provide several compression techniques that can compress a G-buffer by exploring data correlations between pixels. These compression techniques can be used by any video game, such as video games that require real-time generated image compression, or video games that use a deferred shading pipeline. Although the compression techniques in this disclosure are described using a deferred shading pipeline, these techniques can be used in other suitable applications, such as applications that involve real-time 2D image compression / decompression.

[0031] Figure 1 shows a block diagram of a buffering module (101) according to several embodiments of the present disclosure. The buffering module (101) can be used in any suitable graphics processing system for geometry buffering to manage a set of per-pixel geometry information. The buffering module (101) includes a compression module (120), a compressed G buffer (110), and a decompression module (130). The buffering module (101) can take a pixel texture as input, compress the texture, and buffer the pixel texture in a compressed format. Furthermore, when the output of the buffering module (101) is accessed, the buffering module (101) can decompress the buffered pixel texture and output the restored (reconstructed) texture of the pixel. In some examples, the buffering module (101) is implemented in software, and the compression module (120) and decompression module (130) are implemented as software instructions that can be executed by a processing circuit.

[0032] According to one aspect of this disclosure, the buffering module (101) may operate at the pixel group level (multiple pixels in a group) rather than the pixel level (single pixel), and may leverage data correlations within each pixel group. For example, the buffering module (101) may operate at the quad level, including 2x2 pixels, and may leverage data correlations between pixels within the quad. In some video game examples, a frame displayed on the screen may contain millions of pixels, but the number of objects may be only a few thousand, so it is very likely that adjacent pixels share the same object / texture settings.

[0033] It should be noted that in some examples, leveraging more pixel data correlation is likely to make the compression algorithm work more efficiently without sacrificing quality. In some examples, larger pixel groups, such as 4x4, can be used to improve compression efficiency. According to one aspect of this disclosure, some 3D graphics acceleration hardware operates at quad-level (2x2 pixels), and using quad-level can reduce algorithmic complexity. In some examples, larger groups (larger than quad) can improve compression efficiency, but larger groups may require a significant increase in algorithmic complexity, particularly on low-power devices (e.g., mobile devices), where real-time processing may no longer be feasible.

[0034] According to one aspect of the present disclosure, a compression module (120) is configured to sample one or more texture components for pixels in space for compression. In some examples, the compression module (120) samples texture components for pixels in space for compression, for example, using a checkerboard pattern. Furthermore, the compression module (120) can sample two texture components for a pixel using complementary checkerboard patterns, so that the two texture components for a pixel are stored in the original memory space for one texture component of the pixel, and a 2:1 compression ratio can be achieved.

[0035] In some embodiments, to compress the RGB color of a pixel (including the R, G, and B components), the RGB color of each pixel is losslessly converted to the YCoCg color space and then converted to a lumen (Y) component (higher sensitivity to the human eye) and two chroma (Co and Cg) components (lower sensitivity to the human eye). In some examples, the Y component of a pixel is buffered at full spatial resolution in a G-buffer channel that can retain all lumen detail, while the chroma component is sampled by a complementary checkerboard pattern at half spatial resolution to achieve a 2:1 compression ratio and buffered in the G-buffer channel.

[0036] Figure 2 shows diagrams of color compression in several examples. In the examples in Figure 2, the color of a quad (2x2 pixels) includes the quad's Y component (201), quad's Co component (202), and quad's Cg component (203). The quad contains four pixels, indicated by 0, 1, 2, and 3. The quad's Y component (201) includes the values ​​Y_0, Y_1, Y_2, and Y_3; the quad's Co component (202) includes Co_0, Co_1, Co_2, and Co_3; and the quad's Cg component (203) includes Cg_0, Cg_1, Cg_2, and Cg_3.

[0037] The compression module (120) can provide the Y component (201) of the quad (211) to the G buffer channel (e.g., the Y component) with full spatial resolution as a buffered quad (211), thus enabling the preservation of all lumens detail.

[0038] The compression module (120) can sample chroma components such as the Co component (202) and Cg component (203) with half spatial resolution using a complementary checkerboard pattern. The compression module (120) can assemble the sampled data into a mixed quad (212) and provide the mixed quad to a G buffer channel (for example, for the compressed chroma component) for buffering. The compression module (120) achieves a 2:1 compression ratio for the chroma component. The mixed quad (212) contains Co_0 and Co_3 of the Co component (202) and Cg_1 and Cg_3 of the Cg component (203).

[0039] It should be noted that the compression module (120) operates at the quad level rather than the pixel level, achieving various advantages over related examples. For example, since the G-buffer channel can have the full numerical resolution (e.g., 8 bits) as a texture of the input pixels, there are no data quantization errors for compression and no color banding artifacts are produced by the compression. Furthermore, the compressed data can be stored in the G-buffer channel without any bit packing (e.g., packing two 4-bit channels into one 8-bit channel), and thus the use of the compression module (120) enables fully hardware-accelerated framebuffer operations, such as blending and accumulation. It should be noted that bit packing may rely on software solutions that first decompress and expand the packed channels within the shader, then blend them, and repack the blended bits into the compressed channel. Software solutions consume arithmetic logic unit (ALU) cycles, are slower than hardware circuits, and consume much more power than hardware circuits. Since decals are blended with the G-buffer channel after the main scene has been rendered, the absence of bitpacking can be crucial in video games for effects that heavily rely on G-buffer blending, such as "delayed decals." Furthermore, operating at the quad level can enable the use of data correlation in compression. For example, leveraging data correlation can result in better quality at the same compression ratio, or better compression ratio at the same quality.

[0040] Please note that the compression in Figure 2 is for illustrative purposes only and can be modified as appropriate. For example, the sampling patterns of the two chroma components can be switched.

[0041] The decompression module (130) is configured to perform quad-level decompression. For texture components buffered in a checkerboard pattern, the decompression module (130) may determine the value of pixels where texture component information is missing based on the texture component information of adjacent pixels.

[0042] Note that mobile GPU and API limitations may make accessing adjacent pixels difficult. In some cases, APIs are limited to accessing a single pixel. In some cases, mobile GPUs use a tile-based rendering architecture. A tile-based rendering architecture divides the screen into many tiles and renders each tile one by one using on-chip memory (e.g., static random access memory on the same chip as the GPU). Rendered tiles are written to external memory, such as dynamic random access memory (DRAM). When operation primarily utilizes on-chip memory, external bandwidth requirements are significantly reduced, power consumption is lower, and the mobile user experience can be improved. In a tile-based rendering architecture, if a pixel is on a tile boundary, the immediately adjacent pixel may be outside the tile and not available in on-chip memory. In some cases, APIs such as framebuffer fetch, Vulkan subpass, and pixel local storage severely limit framebuffer access to the current pixel coordinates.

[0043] However, according to one aspect of this disclosure, the GPU's internal pipeline for pixel processing can process pixels using quads as the basic unit, rather than individual pixels. The GPU's internal pipeline for pixel processing requires gradient information for various purposes, such as depth slope and mipmap calculations, which can be computed very efficiently with the help of adjacent pixels.

[0044] In some examples, in tile-based rendering architectures, the tile size is generally a multiple of 2. Therefore, even pixels on the tile boundary will not cross the quad as long as the starting coordinates of the current fullscreen primitive are a multiple of 2. In various implementations, all quads on the GPU are aligned to even pixel coordinates.

[0045] In some embodiments, the full-screen primitive is set to a single oversized triangle that completely covers the screen.

[0046] Figure 3 shows the screen (310) and primitive (320) in several examples. The primitive (320) is set as a single oversized triangle that completely covers the screen (310). The start of the primitive (320) is set to 0. Thus, it can be ensured that the quad starts at an even pixel coordinate. Note that in some related examples, the primitive is set as two smaller triangles that cover the screen, and therefore some quads may have diagonals that cross the viewport where odd coordinates may occur.

[0047] In some embodiments, the GPU may have infrastructure to support quad-based decompression and may have several intrinsic instructions that can perform inter-pixel operations. Intrinsic instructions that can perform inter-pixel operations can be used to support quad-based decompression. In some examples, the GPU may operate on a quad to generate inter-pixel gradients (finite differences) in response to several intrinsic instructions (also called inter-pixel gradient intrinsic instructions). The texture values ​​of adjacent pixels can then be obtained using the inter-pixel gradients and an inverse calculation of the texture value of the current pixel. As an example, the intrinsic functions ddx, ddx_coarse, and ddx_fine can return partial derivatives with respect to the screen space x coordinate, and the intrinsic functions ddy, ddy_coarse, and ddy_fine can return partial derivatives with respect to the screen space y coordinate. Thus, the texture values ​​of adjacent pixels can be obtained based on the intrinsic functions (e.g., ddx, ddx_coarse, ddx_fine, ddy, ddy_coarse, ddy_fine) and the texture value of the current pixel. It should be noted that GPUs generally support quad-based arithmetic and inter-pixel gradient intrinsic instructions. Therefore, decompression using quad-based arithmetic and inter-pixel gradient intrinsic instructions can be performed by most GPUs, and thus decompression using quad-based arithmetic and inter-pixel gradient intrinsic instructions can be cross-platform and cross-vendor.

[0048] In some embodiments, the decompression module (130) can decode a checkerboard-like interleaved quad by filling in missing data according to the nearest adjacent pixel.

[0049] Note that filing data using adjacent pixels in a fixed direction (for example, adjacent pixels horizontally (or vertically)) can result in artifacts, particularly in adjacent pixel areas.

[0050] According to one aspect of the present disclosure, the decompression module (130) can select the optimal adjacent pixels by utilizing the inter-pixel gradient. The compressed G-buffer (110) buffers the texture components of the pixels in the checkerboard pattern, and for missing data of each pixel, two nearest adjacent pixels, such as horizontally adjacent pixels and vertically adjacent pixels, can provide the missing data. The decompression module (130) can leverage this flexibility to have a much better opportunity to select the most suitable missing data rather than a predefined one that leads to the aforementioned artifact.

[0051] In some examples, the user can set a numerical local function f(x,y) that defines the likelihood of surface properties. For example, the numerical local function f(x,y) can be set based on the current pixel depth, and the numerical local function f(x,y) can indicate geometric discontinuities. In another example, the numerical local function f(x,y) can be set based on a combination of the current pixel depth and material (e.g., Material_ID), and thus the numerical local function f(x,y) can be used to identify the best-fitting adjacent pixel with extra information about material discontinuities. In some examples, after the numerical local function f(x,y) has been evaluated for the current pixel, the gradient of the numerical position function, such as the partial derivative with respect to the x coordinate (e.g., df(x,y) / dx) and the partial derivative with respect to the y coordinate (df(x,y) / dy), can be determined using, for example, the inter-pixel gradient built-in instruction. The partial derivative with respect to the x coordinate and the partial derivative with respect to the y coordinate are compared. Then, based on the comparison, either the horizontally adjacent pixels or the vertically adjacent pixels can be selected to fill in the missing data. For example, a comparison can show a small gradient from the partial derivative of a numerical local function, and therefore, adjacent pixels along the small gradient can be selected to fill in the missing data of the current pixel. Thus, the missing data is determined from the most likely adjacent pixels that belong to the same geometry or material instance as the current pixel. Therefore, a higher quality reconstruction of the missing data can be achieved.

[0052] Figure 4 illustrates color decompression in several examples. In the examples in Figure 4, the color of a quad (2x2 pixels) is buffered as a first buffered quad (411) in a first G-buffer channel (e.g., for the Y component) and a second buffered quad (412) in a second G-buffer channel (e.g., for the mixed Co and Cg components). The first buffered quad (411) is similar to the buffered quad (211) in Figure 2 and contains rumor information with full spatial resolution, preserving all rumor details. The second buffered quad (412) is similar to the mixed quad (212) in Figure 2 and contains complementary checkerboard pattern Co and Cg information.

[0053] In some examples, the decompression module (130) can provide a first buffered quad (411) as the Y component (421) of the reconstructed quad. Furthermore, the decompression module (130) can reconstruct the Co component (422) and Cg component (423) of the reconstructed quad based on a second buffered quad (412) in a second G buffer channel.

[0054] Using the Co information for pixel 1 as an example, the decompression module (130) can determine Co_1' based on Co_0 and Co_3. For example, the decompression module (130) can use, for example, an inter-pixel gradient built-in instruction to evaluate the gradient of a numerical position function (e.g., a combination of depth and material) for the current pixel (pixel 1) which has missing information, such as partial derivatives with respect to the x-coordinate and partial derivatives with respect to the y-coordinate. For example, the decompression module (130) compares the partial derivative of the numerical position function with respect to the x-coordinate and the partial derivative of the numerical position function with respect to the y-coordinate to determine which of the two partial derivatives is smaller. If the partial derivative with respect to the x-coordinate is smaller than the partial derivative with respect to the y-coordinate, the decompression module (130) can determine Co_1' based on Co_0, and if the partial derivative with respect to the y-coordinate is smaller than the partial derivative with respect to the x-coordinate, the decompression module (130) can determine Co_1' based on Co_3.

[0055] Similarly, the decompression module (130) can determine Co_2' based on Co_0 and Co_3, determine Cg_0' based on Cg_1 and Cg_2, and determine Cg_3' based on Cg_1 and Cg_2.

[0056] Figure 5 shows an image diagram for comparing the results of a related compression example and a G-buffer compression example of this disclosure. In Figure 5, the first image (501) is the original image, the second image (510) is a reconstructed image based on the related chroma subsampling technique, and the third image (520) is a reconstructed image based on the G-buffer compression technique disclosed herein. In one example, the related chroma subsampling technique uses 4:2:2 subsampling, which samples the chroma component with half horizontal resolution and full vertical resolution. Therefore, during reconstruction, missing chroma values ​​are filled in by adjacent pixels in a certain direction, such as the horizontal.

[0057] As shown in Figure 5, the second image (510) contains artifacts introduced by the associated chroma subsampling technique. The third image (520) does not have the artifacts that appeared in the second image (510). The G-buffer compression technique achieves higher quality reconstruction of chroma information than the associated chroma subsampling technique.

[0058] It should be noted that G-buffer compression techniques can be used with any appropriate texture component and with any appropriate graphics processing pipeline.

[0059] Figure 6 shows a diagram of a graphics processing pipeline (600) according to several embodiments of the present disclosure. According to one embodiment, the graphics processing pipeline (600) is configured for delayed shading. In some examples, the graphics processing pipeline (600) is implemented with software instructions that can be executed by one or more GPUs to perform graphics processing operations.

[0060] In the example in Figure 6, the graphics processing pipeline (600) includes a first pass (640) and a second pass (650). The first pass (640) includes, for example, a compression module (120) at its output, and the second pass (650) includes, for example, a decompression module (130) at its input. The first pass (640) can generate pixel geometry information and compress it for buffering into a compressed G-buffer (110). The second pass (650) can access the compressed G-buffer (110), reconstruct the pixel geometry information, and perform additional graphics processing operations to generate, for example, a final image.

[0061] In some examples, the first pass (640) may include several processing modules, such as a collection module (641) for collecting pixel geometry information, and the second pass (650) may also include several processing modules, such as a vertex shading module (651) and a fragment shading module (655). In some embodiments, the vertex shading module (651) and fragment shading module (655) of the second pass (650) may be referred to as lazy shading.

[0062] In some embodiments, the collection module (641) can perform appropriate geometry calculations to determine per-pixel geometry information without considering lighting. This geometry information may be required for subsequent lighting calculations. The vertex shading module (651) and fragment shading module (655) in the second pass (650) can perform further processing such as shading, lighting, and shadow calculations. In some examples, the vertex shading module (651) and fragment shading module (655) in the second pass (650) can perform shading calculations on the parts affected by lighting.

[0063] According to one aspect of this disclosure, when pixel geometry information is buffered with an interleaved pattern such as a checkerboard pattern, not only can the memory space of the texture component be amortized, but the computational cost can also be amortized between pixels.

[0064] In some examples, a technique called cooperative computing is used in a second pass for decompression, shading, etc. For example, shading the current pixel may involve operations on the geometry information of adjacent pixels. In one example (called the gather-compute example), the geometry information of adjacent pixels is collected for decompression or shading of the current pixel, and calculations are performed on the current pixel and adjacent pixels. In the gather-compute example, there may be redundant parts in the calculations for the current pixel and adjacent pixels.

[0065] Figure 7 shows the shader's behavior for two adjacent pixels (pixel A and pixel B) in a gather-compute example. For shading pixel A, the buffered geometry information for pixel A is available, and the operation to calculate the rendering effect of pixel A is shown as (710). For shading pixel B, the buffered geometry information for pixel B is available, and the operation to calculate the rendering effect of pixel B is shown as (720). In some examples, operations (710) and (720) are executed in parallel. As shown in (710) and (720), operations 2B-3B are redundant to operations 2A-3A.

[0066] In another example (called the compute-gather example), operations are performed on buffered geometry information of individual pixels to generate an intermediate result. The intermediate results are then exchanged, and the rendering effect calculations may be performed.

[0067] Figure 8 shows the shader behavior for two adjacent pixels (pixel A and pixel B) in a compute-gather example. For shading pixel A, buffered geometry information for pixel A is available, and the operation to calculate the rendering effect of pixel A is shown as (810). For shading pixel B, buffered geometry information for pixel B is available, and the operation to calculate the rendering effect of pixel B is shown as (820).

[0068] In some examples, the computation in the compute-gather example is reduced to approximately one-third compared to the gather-compute example.

[0069] The G-buffer compression techniques in this disclosure can offer various advantages. For example, the G-buffer compression techniques in this disclosure can support G-buffer compression with virtually no loss of quality, saving bandwidth and memory space. Furthermore, the G-buffer compression techniques in this disclosure can support hardware-accelerated G-buffer blending, which can be used to produce certain effects such as decal rendering. In addition, the G-buffer compression techniques in this disclosure can improve the tile-based rendering performance of mobile GPUs. For example, by reducing the memory storage overhead per tile, the number of pixels per tile (bin) can be increased (for example, with the same amount of on-chip memory), and the total number of tiles for the same resolution can be decreased. As a result, the workload on vertices is reduced, and the parallelism of pixels in the processing of each tile is improved. In one example, in a mobile architecture, when the G-buffer compression techniques in this disclosure are applied, the number of pixels per tile can increase from 240×192 to 240×288, and the total number of tiles per frame can decrease from 27 to 18.

[0070] Furthermore, in some examples, when the G-buffer compression techniques described herein are applied, algorithms may be used that amortize the computation with minimized instruction overhead.

[0071] Figure 9 shows a flowchart illustrating an overview of process (900) according to one embodiment of the present disclosure. In some embodiments, process (900) is implemented as a software instruction, and therefore, when a processing circuit executes a software instruction, the processing circuit executes process (900). The process starts at (S901) and proceeds to (S910).

[0072] In (S910), compressed geometry information for buffering in a geometry buffer (G-buffer) is generated based on the geometry information of the pixels in the frame. The compressed geometry information includes a first texture component for a first group of pixels, omitting the first texture component for at least one second pixel, where the second pixel has one or more adjacent pixels in the first group of pixels. The compressed geometry information is buffered in G-buffers within multiple channels (also called G-buffer channels). Each channel buffers information for one or more texture components.

[0073] In (S920), the geometry buffer is accessed to obtain compressed geometry information.

[0074] In (S930), reconstructed pixel geometry information is generated based on the compressed geometry information. The reconstructed geometry information includes a first texture component for a second pixel that is reconstructed based on a first texture component of one or more adjacent pixels in the first group of pixels.

[0075] In (S940), rendering is performed to generate an image according to the reconfigured geometry information of the pixels.

[0076] According to one aspect of the present disclosure, compressed geometry information includes a first texture component for a first group of pixels and a second texture component for a second group of pixels. The first group of pixels has a first checkerboard pattern in screen space (e.g., pixels displayed on the screen), and the second group of pixels has a second checkerboard pattern complementary to the first checkerboard pattern. As an example, a combination of the first and second groups of pixels can completely fill screen space. In some examples, the first texture component for the first group of pixels and the second texture component for the second group of pixels are buffered in channels of a geometry buffer. Each entry in the channels of the geometry buffer corresponds to a pixel in screen space and stores a value for either the first texture component for the pixel or the second texture component for the pixel.

[0077] In some examples, the compression and decompression of geometry information is performed by a quad. In some examples, for a 2x2 pixel quad containing two first pixels in the first diagonal direction and two second pixels in the second diagonal direction, the first texture component of the first two pixels and the second texture component of the second two pixels may be buffered in channel space for the quad in a channel of the geometry buffer to generate compressed geometry information (e.g., compressed). Furthermore, in some examples, to generate reconstructed geometry information (e.g., decompressed), the second texture component of the first two pixels is determined based on the second texture component of the second two pixels, and the first texture component of the second two pixels is determined based on the first texture component of the first two pixels.

[0078] In some embodiments, to ensure that the quads start at even pixel coordinates, the fullscreen primitive is defined as a triangle that covers the screen space for displaying the image. Thus, compression and decompression can be safely performed by quads that start at even pixel coordinates.

[0079] In some embodiments, shader operations can be amortized by cooperative computation. For example, pixels can be rendered in parallel by multiple processing units to perform rendering. In one example, a first processing unit can perform a first operation on a first texture component for pixels in a first pair of pixels to produce a first intermediate result, and a second processing unit can perform a second operation on a second texture component for pixels in a second pair of pixels to produce a second intermediate result. The first and second intermediate results are exchanged between the first and second processing units. Based on the first and second intermediate results, the first processing unit can perform rendering for pixels in the first pair of pixels. Based on the first and second intermediate results, the second processing unit can perform rendering for pixels in the second pair of pixels.

[0080] In some examples, the second pixel has a first adjacent pixel along a first direction (e.g., either horizontal or vertical) and a second adjacent pixel along a second direction (e.g., the other of horizontal and vertical). To generate reconstructed geometric information, in some examples, the direction in which the function at the second pixel has a smaller gradient can be determined from the first and second directions. The first texture component for the second pixel can then be reconstructed based on one of the first and second adjacent pixels along that direction. This function can be a predefined function to indicate the continuity or discontinuity of the surface properties. The surface properties may include at least one of depth properties and material properties.

[0081] The first texture component can be any suitable texture component, such as a color component (for example, one of the chroma components of color information), a surface normal, or a surface depth.

[0082] In some examples, the first texture component for a second pixel can be obtained by performing an inverse calculation of the inter-pixel gradient to reconstruct the first texture component for the second pixel.

[0083] Next, the process proceeds to (S999) and terminates.

[0084] Process (900) may be appropriately adapted. The steps of Process (900) may be modified and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0085] The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, Figure 10 shows a computer system (1000) suitable for implementing a particular embodiment of the disclosed subject matter.

[0086] Computer software can be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that contains instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0087] Instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0088] The components of the computer system (1000) shown in Figure 10 are essentially illustrative and are not intended to imply any limitations on the scope or functionality of computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependencies or requirements on any component or combination of components shown in the exemplary embodiments of the computer system (1000).

[0089] The computer system (1000) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users through, for example, haptic input (keystrokes, swipes, data glove movements, etc.), voice input (voice, applause, etc.), visual input (gestures, etc.), olfactory input (not shown), etc. The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as voice (voices, music, ambient sounds, etc.), images (scanned images, photographic images taken from a still camera, etc.), and videos (2D videos, 3D videos including stereoscopic videos, etc.).

[0090] The input human interface device may include one or more of the following (only one of each is shown): a keyboard (1001), a mouse (1002), a trackpad (1003), a touchscreen (1010), a data glove (not shown), a joystick (1005), a microphone (1006), a scanner (1007), and a camera (1008).

[0091] The computer system (1000) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, smell / taste, etc. Such human interface output devices may include tactile output devices (for example, touchscreens (1010), data gloves (not shown), or tactile feedback via joysticks (1005), although tactile feedback devices that do not function as input devices may also exist), audio output devices (for example, speakers (1009), headphones (not shown)), visual output devices (screens (1010), including CRT screens, LCD screens, plasma screens, OLED screens, etc., each of which may or may not have touchscreen input functionality, each of which may or may not have tactile feedback functionality, and some of which are capable of 2D visual output or output beyond 3D by means such as stereographic output, virtual reality glasses (not shown), holographic displays and smoke tanks (not shown)), and printers (not shown).

[0092] The computer system (1000) may also include human-accessible storage devices and associated media, such as optical media (1021) including CD / DVD ROM / RW (1020), thumb drives (1022), removable hard drives or solid-state drives (1023), legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0093] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include a transmission medium, carrier wave, or other transient signal.

[0094] The computer system (1000) may also include an interface (1054) to one or more communication networks (1055). The networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicle, industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet, cellular networks including Wi-Fi, GSM, 3G, 4G, 5G, and LTE, wide-area digital networks for wired or wireless TV including cable TV, satellite TV, and terrestrial broadcast TV, and vehicle and industrial networks including CANBus. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (1049) (e.g., a USB port on the computer system (1000)), while others are typically integrated into the core of the computer system (1000) by being attached to a system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1000) can communicate with other entities. Such communication can be unidirectional and receivable only (e.g., television broadcasting), unidirectional and receivable only (e.g., from CANbus to a specific CANbus device), or bidirectional, for example, with other computer systems using local or wide-area digital networks. As described above, specific protocols and protocol stacks may be used on each network and network interface.

[0095] The aforementioned human interface device, human-accessible memory device, and network interface can be attached to the core (1040) of the computer system (1000).

[0096] The core (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), specialized programmable processing units in the form of field-programmable gate areas (FPGAs) (1043), hardware accelerators for specific tasks (1044), graphics adapters (1050), and the like. These devices may be connected via a system bus (1048) along with read-only memory (ROM) (1045), random access memory (1046), and internal mass storage devices such as built-in hard drives and SSDs (1047) that are not accessible to the user. In some computer systems, the system bus (1048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (1048) or via a peripheral bus (1049). For example, a screen (1010) may be connected to a graphics adapter (1050). Peripheral bus architectures include PCI and USB.

[0097] The CPU (1041), GPU (1042), FPGA (1043), and accelerator (1044) can execute specific instructions that can be combined to form the aforementioned computer code. This computer code can be stored in ROM (1045) or RAM (1046). Transition data can also be stored in RAM (1046), while permanent data can be stored, for example, in internal mass storage (1047). By using cache memory that can be closely associated with one or more CPUs (1041), GPUs (1042), mass storage (1047), ROM (1045), RAM (1046), etc., high-speed storage and retrieval to any of the memory devices can be enabled.

[0098] A computer-readable medium may contain computer code for performing various computer implementation operations. The medium and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0099] For example, but not limited to, a computer system (1000) having an architecture, specifically a core (1040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage described above, or may be specific storage of the core (1040) having a non-transient nature, such as a mass storage (1047) or ROM (1045) inside the core. Software implementing various embodiments of this disclosure may be stored in such devices and executed by the core (1040). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software may cause the core (1040), specifically the processor (including a CPU, GPU, FPGA, etc.) within it, to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1046) and modifying such data structures according to processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logic embodied in a circuit (e.g., an accelerator (1044)) in a hardwired or otherwise manner, which may operate in place of or with software to perform a particular process or a particular part of a particular process described herein. Where appropriate, references to software may encompass logic and vice versa. References to a computer-readable medium may, where appropriate, encompass a circuit (such as an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both. This disclosure encompasses any appropriate combination of hardware and software.

[0100] While this disclosure has described several exemplary embodiments, there are many modifications, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, those skilled in the art will understand that numerous systems and methods, not expressly shown or described herein, can be devised to embody the principles of this disclosure and fall within its spirit and scope. [Explanation of Symbols]

[0101] 101 Buffering Module 110 Compressed G-buffer 120 Compression Modules 130 Decompression Module Component 201 Y 202 Co component 203 Cg component 211 Buffered Quad 212 Mixed Quad 310 screens 320 Primitives 411 First buffered quad 412 Second buffered quad 421 Reconstructed Quad Y component 422 Reconstructed Quad Co component 423 Reconstructed Quad Cg component 501 Image 1 510 Second image 520 Third image 600 graphics processing pipelines 640 First Pass 641 Collection Module 650 Second Pass 651 Vertex Shading Module 655 Fragment Shading Module 1000 Computer Systems 1001 Keyboard 1002 Mouse 1003 Trackpad 1005 Joystick 1006 Microphone 1007 Scanner 1008 Camera 1009 Speaker 1010 Touchscreen 1020 CD / DVD ROM / RW 1021 Optical media 1022 Thumb Drive 1023 Removable hard drive or solid state drive 1040 cores 1041 Central Processing Unit CPU 1042 Graphics Processing Units (GPUs) 1043 Field-Programmable Gate Area FPGA 1044 Hardware Accelerators 1045 Read-only memory ROM 1046 random access memory 1047 Internal mass storage 1048 System Bus 1049 General-purpose data port or peripheral bus 1050 Graphics Adapter 1054 Interface 1055 Communication Network

Claims

1. A step of generating compressed geometry information for buffering in a geometry buffer (G buffer) based on the geometry information of pixels in a frame, wherein the compressed geometry information includes a first texture component for a first group of pixels, omits the first texture component for at least one second pixel, and the second pixel has one or more adjacent pixels in the first group of pixels. To obtain the compressed geometry information, the steps include: accessing the geometry buffer; A step of generating reconstructed geometry information of the pixels based on the compressed geometry information, wherein the reconstructed geometry information includes the first texture component for the second pixel reconstructed based on the first texture component of one or more adjacent pixels in the first group of pixels, The steps include: performing rendering to generate an image according to the reconstructed geometry information of the pixels; A method for image rendering, including...

2. The method according to claim 1, wherein the compressed geometry information includes a first texture component for the first group of pixels and a second texture component for the second group of pixels, wherein the first group of pixels has a first checkerboard pattern and the second group of pixels has a second checkerboard pattern complementary to the first checkerboard pattern.

3. Steps to buffer the first texture component for the first group of pixels and the second texture component for the second group of pixels within the channels of the geometry buffer. The method according to claim 2, further comprising:

4. The step of generating the compressed geometry information for a 2x2 pixel quad including two first pixels in the first diagonal direction and two second pixels in the second diagonal direction is: The step of buffering the first texture components of the first two pixels and the second texture components of the second two pixels into the channel space for the quad. The method according to claim 2, including the method described in claim 2.

5. The step of generating the reconstructed geometry information is, A step of determining the second texture component of the first two pixels based on the second texture component of the second two pixels, A step of determining the first texture component of the second two pixels based on the first texture component of the first two pixels. The method according to claim 4, further comprising:

6. The steps include defining a fullscreen primitive as a triangle that covers the screen space for displaying the image, The steps include generating the reconstructed geometry information using quads that begin with even pixel coordinates, and The method according to claim 5, further comprising:

7. The step of performing the rendering described above is, To generate a first intermediate result, a first processing unit performs a first operation on the first texture component for one pixel in the first two pixels, To generate a second intermediate result, the second processing unit performs a second operation on the second texture component for one of the two second pixels, A step of exchanging the first intermediate result and the second intermediate result between the first processing unit and the second processing unit, The first processing unit performs rendering on the pixels within the first two pixels based on the first intermediate result and the second intermediate result, The second processing unit performs rendering for the pixels in the two second pixels based on the first and second intermediate results. The method according to claim 4, further comprising:

8. The second pixel has a first adjacent pixel along the first direction and a second adjacent pixel along the second direction, and the step of generating the reconstructed geometry information is, The steps of determining the direction in which the function has a smaller gradient from the first direction and the second direction, A step of reconstructing the first texture component for the second pixel based on one of the first adjacent pixels and the second adjacent pixels along the aforementioned direction. The method according to claim 1, further comprising:

9. The step of reconstructing the first texture component for the second pixel is The step of performing an inverse calculation of the inter-pixel gradient in order to obtain the first texture component of the second pixel. The method according to claim 8, further comprising:

10. The method according to claim 8, wherein the function is defined in advance to indicate a discontinuity in the surface properties.

11. The method according to claim 10, wherein the surface properties include at least one of depth properties and material properties.

12. The method according to claim 1, wherein the first texture component includes at least one of a color component, a surface normal, and a surface depth.

13. The method for generating compressed geometry information for buffering in a geometry buffer (G-buffer) based on the geometry information of pixels within a frame, wherein the compressed geometry information includes a first texture component for a first group of pixels, omits the first texture component for at least one second pixel, and the second pixel has one or more adjacent pixels in the first group of pixels. To obtain the compressed geometry information, access the geometry buffer, To generate reconstructed geometry information of the pixels based on the compressed geometry information, wherein the reconstructed geometry information includes the first texture component for the second pixel reconstructed based on the first texture component of one or more adjacent pixels in the first pixel group. Performing rendering to generate an image according to the reconstructed geometry information of the aforementioned pixels A device comprising a processing circuit configured to perform the following actions.

14. The apparatus according to claim 13, wherein the compressed geometry information includes a first texture component for the first group of pixels and a second texture component for the second group of pixels, the first group of pixels having a first checkerboard pattern and the second group of pixels having a second checkerboard pattern complementary to the first checkerboard pattern.

15. The processing circuit described above The first texture component for the first group of pixels and the second texture component for the second group of pixels are buffered within the channels of the geometry buffer. The apparatus according to claim 14, configured as follows.

16. For a 2x2 pixel quad including two first pixels in the first diagonal direction and two second pixels in the second diagonal direction, the processing circuit: The first texture components of the first two pixels and the second texture components of the second two pixels are buffered in the channel space for the quad. The apparatus according to claim 14, configured as follows.

17. The processing circuit described above Based on the second texture components of the two second pixels, the second texture components of the two first pixels are determined, Based on the first texture components of the first two pixels, the first texture components of the second two pixels are determined. The apparatus according to claim 16, configured to perform the following:

18. The processing circuit described above A fullscreen primitive is defined as a triangle that covers the screen space for displaying the aforementioned image, The reconstructed geometry information is generated by a quad that starts with an even pixel coordinate. The apparatus according to claim 17, configured to perform the following:

19. The processing circuit comprises a first processing unit and a second processing unit, and the processing circuit is To generate a first intermediate result, the first processing unit performs a first operation on the first texture component for one pixel in the first two pixels, To generate a second intermediate result, the second processing unit performs a second operation on the second texture component for one of the two second pixels, The first intermediate result and the second intermediate result are exchanged between the first processing unit and the second processing unit. The first processing unit performs rendering on the pixels within the first two pixels based on the first intermediate result and the second intermediate result, The second processing unit performs rendering on the pixels within the two second pixels based on the first and second intermediate results. The apparatus according to claim 16, configured to perform the following:

20. The second pixel has a first adjacent pixel along the first direction and a second adjacent pixel along the second direction, and the processing circuit, From the first and second directions, determine the direction in which the function has a smaller gradient, Reconstructing the first texture component for the second pixel based on one of the first adjacent pixels and the second adjacent pixels along the aforementioned direction. The apparatus according to claim 13, configured to perform the following:

Images