SYSTEMS AND METHODS FOR ENCODER-GUIDED ADAPTIVE QUALITY RENDERING

Abstract

Systems and methods are disclosed for improving computer technology related to image rendering and encoding, preferably for use in a video game environment. In certain embodiments, a codec is used to encode one or more reference images for a partial range of encoder settings, and a renderer is used to generate one or more rendering quality setting profiles, generate one or more reference images, calculate the perceived qualities for each of the reference images, re-render one or more reference images for each of the rendering quality setting profiles, and calculate the perceived qualities for each of the rendered reference images. The renderer compares the perceived qualities of the reference images with the perceived qualities of the re-rendered images and combines them.These matches result in associating one or more encoder settings with their corresponding render quality settings profiles in a lookup table.

Description

1 / 21 “SYSTEMS AND METHODS FOR RENDERING "CODER-GUIDED ADAPTIVE QUALITY" DESCRIPTIVE REPORT RELATED ORDERS

[0001] This application claims the benefit of the following U.S. provisional applications: 62 / 488,526, filed April 21, 2017, and 62 / 653,056, filed April 5, 2018. BACKGROUND OF THE INVENTION

[0002] Remote gaming applications, in which a server-side game is controlled by a client-side player, have attempted to encode the video output of a real-time three-dimensional (3D) graphics engine using existing or custom codecs, also known as encoders. However, the interactive nature of video games, particularly the player feedback loop between video output and player input, makes game video streaming far more latency-sensitive than traditional video streaming. Existing video encoding methods may trade off computational power, and little else, for reductions in encoding time.New methods for integrating the encoding process with the video rendering process can provide significant reductions in encoding time, as well as reduce computational power, improve the quality of the encoded video, and maintain the original bitstream data format to preserve the interoperability of existing hardware devices.

[0003] When a video game instance is running on local hardware for the player, it is desirable for the game to render every pixel Petition 870190104166, dated 10 / 15 / 2019, page 5 / 119 2 / 21 with the highest quality. However, in a server-side game instance where the rendered output is encoded and streamed to a remote client, the encoder may reduce the image quality to fit within a limited bandwidth. If the rendered quality is much higher than the decoded output quality, a measurable amount of server-side rendering work will be lost.

[0004] By adapting the server-side rendered quality to match the post-quantization quality based on encoder feedback, the game can reduce server-side computational waste without any perceptible loss of quality on the client side. The reduction in server-side computational waste can also result in additional benefits, including reduced power usage, reduced rendering times, and reduced player feedback latency. Server-side computational savings are compounded in environments where multiple game instances are running on the same server.

[0005] In streaming environments for games involving multiple players, particularly games like Massive Multiplayer Online Games (MMOGs), ensuring that server-side rendering work is not wasted becomes increasingly important. Due to the limited bandwidth available to MMOG players, an encoder that maximizes rendering quality while avoiding game slowdown is particularly important. Current technologies, as discussed below, employ various methods to try and solve this problem, but remain deficient.

[0006] US Patent Publication No. US20170132830A1 (“a Publication '830' discloses systems and methods for determining a selected shading point in a 3D scene where shading should be applied. Petition 870190104166, dated 10 / 15 / 2019, page 6 / 119 3 / 21 is executed, performing shading at the determined shading point and determining 3D scene shading information based on a shading result performed at the determined shading point. The scene shading is adjusted based on the temporal characteristics of the scene. However, this technology does not address the fundamental problem of optimizing encoding based on server-side rendering capabilities and available bandwidth.

[0007] U.S. Patent Publication No. US20170200253A1 (“the Publication '253'”) discloses systems and methods for improving the rendering performance of graphics processors. In the graphics processor, an upper limit can be set so that when a frame larger than the defined limit is encountered, the graphics processor takes appropriate measures to reduce rendering time. However, this technology is based only on a defined limit and does not dynamically adjust to server-side rendering capabilities and available bandwidth.

[0008] US Patent Publication No. US2017 / 0278296A1 (“a Publication '296' discloses systems and methods in which an initial rendering of a scene that determines the texture in each part of the scene is generated, and a ray-traced rendering of the scene is generated by tracing an initial sample of rays. This reference discloses that an ideal number of samples for each pixel is intelligently determined based on prior knowledge of the scene's textures and the identification of noise resulting from undersampling during ray tracing. Again, this technology is limited to optimal ray sampling and does not dynamically adjust to server-side rendering capabilities and available bandwidth.

[0009] As is evident from the discussion above about the state Petition 870190104166, dated 10 / 15 / 2019, page 7 / 119 4 / 21 of the technique in this technology, there is a need in the technique to improve the current computer technology related to game rendering and encoding. SUMMARY OF THE INVENTION

[0010] Therefore, it is an object of the present invention to disclose systems and methods for improving rendering using a codec (which may also be referred to herein as an encoder) to encode one or more reference images to a partial range of encoder settings and a renderer to generate one or more rendering quality setting profiles, generate one or more reference images, calculate the perceived qualities for each of the reference images, re-render one or more reference images for each of the rendering quality setting profiles, and calculate the perceived qualities for each of the rendered reference images. The renderer compares the perceived qualities of the reference images with the perceived qualities of the re-rendered images and combines them.These matches result in associating one or more encoder settings with their corresponding render quality settings profiles in a lookup table. The lookup table is used to generate a rendered image with a perceived quality substantially identical to an encoded frame during gameplay.

[0011] Another objective of the present invention is to disclose systems and methods for improving encoding and rendering by using a renderer to calculate computational costs for each of the reference images rendered again.

[0012] It is yet another objective of the present invention to disclose systems and methods for improving encoding and rendering by applying an index. Petition 870190104166, dated 10 / 15 / 2019, page 8 / 119 5 / 21 of structural similarity to calculate the perceived qualities of the images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A fuller appreciation of the invention and many of its corresponding advantages will be readily obtained as it becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, in which:

[0014] FIG. 1 is a diagram of an exemplary environment in which a live streaming codec can communicate settings back to the renderer that produces the video, according to an embodiment of the invention;

[0015] FIG. 2 is a flow diagram that describes exemplary stages of coder-guided adaptive quality rendering according to an embodiment of the invention;

[0016] FIG. 3 is a flow diagram that describes the exemplary pre-generation of the lookup table that assigns a rendering quality settings profile to each partial range of the encoder settings, according to an embodiment of the invention;

[0017] FIG. 4 is a diagram of an exemplary lookup table generation for rendering quality configuration profiles that are composed of only one configuration, according to an embodiment of the invention; and

[0018] FIG. 5 is a diagram of an exemplary lookup table generation for rendering rendering quality configuration profiles containing multiple settings, according to an embodiment of the invention. Petition 870190104166, dated 10 / 15 / 2019, page 9 / 119 6 / 21 DETAILED DESCRIPTION OF THE PREFERRED MODALITIES

[0019] In describing a preferred embodiment of the invention illustrated in the drawings, specific terminology will be used for the sake of clarity. However, the invention is not intended to be limited to the specific terms selected, and it should be understood that each specific term includes all technical equivalents that operate in a similar manner to achieve a similar objective. Several preferred embodiments of the invention are described for illustrative purposes, it being understood that the invention may be carried out in other forms not specifically shown in the drawings.

[0020] Modern rendering engines, such as those used in video games, have the ability to adapt certain quality settings during runtime, based on factors such as the player's distance from an object, the rendering time of the previous frame, or other runtime metrics. A rendering engine can provide various methods for adjusting quality, allowing for more granular control over the overall quality of the rendering. Some examples include polarization texture sampling to use blurry mipmaps, using lower-quality cascades or fewer samples in shadows, performing a simplified path in the shading model (e.g., DCT transformations from specular to appear diffuse), and using fewer samples for post-processing (e.g., for Gaussian, volumetric fog, etc.).In live streaming applications, changing one or more rendering quality settings in response to changes in encoder settings can provide the best rendering cost savings without affecting the quality of the encoded output. Petition 870190104166, dated 10 / 15 / 2019, page 10 / 119 7 / 21

[0021] FIG. 1 is a diagram of an exemplary environment in which real-time rendered video is transmitted live to a remote viewer. The server 100 can be any hardware capable of simultaneously running a real-time rendering process 102 (also known as a renderer below) and a transmission codec 104. The codec 104 must also have the ability to communicate its quantization settings back to the rendering process 102 via direct reporting or some other monitoring process known in the art. The encoded video stream is transmitted over a network to a client device 106. The client 106 can be any hardware capable of decoding and displaying the video stream.

[0022] FIG. 2 is a flowchart that describes the exemplary stages of coder-guided adaptive quality rendering. Live broadcast encoding using an H.264-compliant encoder typically employs a Constant Rate Factor (CRF) mode that reports the effective quantization settings for an encoded frame as a quantization parameter (QP) in “Report quantization settings for each encoded frame,” step 200. In certain modes, the H.264-compliant library used is ffmpeg, which generates the quantization parameter as the variable, f_crf_avg. The quantization parameter is an index, ranging from 0 to 51, that defines the compression loss during encoding. Lower QP values ​​represent less compression, while higher QP values ​​represent greater compression.To maintain a constant bit rate, an encoder operating in CRF mode will increase the QP for frames that can produce greater compression and decrease the QP for frames that require higher quality. The encoder takes advantage of the fact that the human eye is less able to distinguish details of objects in high-resolution frames. Petition 870190104166, dated 10 / 15 / 2019, page 11 / 119 8 / 21 motion, increasing compression in areas with comparatively high motion and decreasing compression in areas that are relatively still. This allows the encoder to maintain a perceived quality of the target while reducing the size of some encoded frames.

[0023] The renderer reads the QP reported before rendering a frame in “Monitor Quantization Settings for Changes,” step 202. In “Different?,” step 203, if the effective quantization settings have not changed since the previously rendered frame, the renderer will not take any action to adapt the rendering quality and will check again on the next frame. If the renderer reads a QP value that is different from the previously rendered frame, or if this is the first encoded frame for which coder-guided adaptive quality rendering is being performed, the rendering quality will be changed in “Change Rendering Quality Settings to Match Quantization Settings,” step 204. If the QP value has increased since the previously rendered frame, the renderer will decrease the quality to match the compression level in the encoder.Similarly, if the Qp value has decreased since the previously rendered frame, the encoder will increase the quality. To change the rendering settings, the renderer will check a pre-generated lookup table that provides a rendering quality settings profile for the Qp value provided by the encoder. In general, there should only be one entry per encoder quality setting. The renderer uses the Qp provided by the encoder, locates the entry, and uses the associated rendering quality settings profile. In general, the entire rendering quality settings profile is applied. A rendering quality settings profile is defined as one. Petition 870190104166, dated 10 / 15 / 2019, page 12 / 119 9 / 21 list of values ​​for each available rendering quality setting. The pre-generation of this lookup table is described in more detail with reference to FIG. 3. The predefined lookup table can define rendering settings for integer QP values, which requires the renderer to round the QP value read to the nearest integer, or the lookup table can define rendering settings for each partial range of QP values ​​between 0 and 51. The examples in FIG. 4 and FIG. 5 assume that the renderer will round the QP to the nearest integer before using the lookup table, but the examples can be modified to define a lookup table using partial QP ranges. The renderer will change the quality settings according to the rendering quality settings profile fetched from the lookup table before rendering the next frame.Reducing the rendering quality will reduce the amount of wasted rendering work when the encoder funnels the quality.

[0024] FIG. 3 is a flowchart that describes the exemplary pre-generation of the lookup table that assigns a rendering quality settings profile to each partial range of the encoder settings. A reference image will be used as a baseline to measure the effects on perceived quality as encoding or rendering settings are changed. The reference image should represent a typical video output frame and include rendered elements such as models, textures, or visual effects typical of a chosen game context. The game context may include a specific area, specific map, specific level, or some specific gameplay. The selected reference image will be used to generate a lookup table that estimates the perceived quality of the rendered video in the same context as the reference image. For example, the table of Petition 870190104166, dated 10 / 15 / 2019, page 13 / 119 A query generated from a reference image containing a representative set of elements from a game level can be used to estimate the perceived quality of video rendered from similar scenes in the same level. Methods for combining multiple lookup tables into a generalized lookup table are discussed further below. After a game context is identified, a representative scene should be chosen and rendered at full quality, as shown in “Select and Generate Reference Image,” step 300. The full-quality rendered scene from the representative scene is referred to here as the reference image.

[0025] A preferred embodiment of the renderer's runtime behavior, discussed above in connection with the description of FIG. 2, requires the renderer to round the received QP values ​​to the nearest integer before reading the lookup table. As a result, the lookup table will be generated using only integer QP values. In the encoder, the full-quality reference image is encoded for each quality setting with an integer value in the encoder, integer quantization parameter (QP) values ​​from 0 to 51, as shown in “Encoding Image of Reference for each Partial Range of Encoder Settings”, step 302. In the preferred mode, there are 52 partial ranges that are defined by the rounding operation performed by the renderer. The implementation can be modified to create more partial ranges for the most common QP values, values ​​in the middle of the range from 0 to 51, or fewer partial ranges for the rarest QP values, values ​​at the extremes of the range from 0 to 51.

[0026] Perceived quality is an attempt to quantify how well the human eye can perceive the loss of quality between a compressed image and the full-quality source image. There are several Petition 870190104166, dated 10 / 15 / 2019, page 14 / 119 11 / 21 methods used to estimate perceived quality include mean squared error (MSE) and peak signal-to-noise ratio (PSNR), which use only the differences in luminance and contrast values ​​between two images to calculate the quality of a compression codec. As reported by Z. Wang, AC Bovik, HR Sheikh, and EP Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, April 2004, the structural similarity index (SSIM) is a method that adds the assumption that the human eye is also an expert at extracting structural information from a scene and defines a calculation to estimate perceived quality. SSIM works by comparing pixel data between two images: the uncompressed full-quality reference image and the encoded image.The algorithm compares luminance, contrast, structure, and sometimes chrominance across 8x8 pixel “windows.” Because SSIM has a low computation cost and outperforms methods like MSE and PSNR, it is the preferred tool for calculating perceived quality. To generate the perceived quality for each value of the encoder settings, preferably in the renderer and / or game engine, the SSIM index is calculated between each encoded reference image and the reference image, as shown in “Calculating Perceived Quality for Each Encoded Reference Image,” step 304. In the preferred embodiment, 52 SSIM values ​​are calculated, one for each integer of the quantization parameter (QP), with a value from 0 to 51. Exemplary descriptions with reference to FIG. 3, FIG. 4, and FIG.5 uses a standard SSIM calculation to compare two still images, but there are variants of the SSIM method that can compare two video segments and that can be used at an increased computational cost. One such variant of SSIM is SSIM. Petition 870190104166, dated 10 / 15 / 2019, page 15 / 119 12 / 21 of Spatio-Temporal, as disclosed by Anush K. Moorthy and Alan C. Bovik, “Efficient Motion Weighted Spatio-Temporal Video SSIM Index,” Human Vision and Electronic Imaging XV, vol. 7527, Mar. 2010 (available at http: / / live.ece.utexas.edu / publications / 2010 / moorthy_spie_jan10.pdf).

[0027] The renderer may have several settings available for pixel-level quality control, including screen resolution, mipmap selection, level of detail (LOD) selection, shadow quality, post-processing quality, or other settings. A quality settings profile is defined as a list of values ​​for each available quality setting. In certain modes, the renderer compiles a list of all adaptively changeable rendering settings, along with their possible values. Then, all permutations of the adaptive quality rendering settings and their values ​​are generated to create a list of rendering quality settings profiles, as shown in “GENERATE List of Rendering Quality Settings Profiles”, step 306.Since a renderer can have many quality settings with many possible values, the number of permutations of quality setting profiles can be prohibitively long. The example in FIG. 5 discusses an exemplary method for limiting and optimizing the number of quality setting profiles in the list.

[0028] For each render quality settings profile in the list, the reference image must be re-rendered in the renderer using the specified render settings, as shown in “Re-render the Reference Image for each Render Quality Settings Profile”, step 308. If the render quality settings profiles consist of more than one setting, the timings of Petition 870190104166, dated 10 / 15 / 2019, page 16 / 119 13 / 21 rendering for each reference image re-rendered should also be recorded as a measure of computational cost, e.g., measured in rendering time or clock cycles. This measure of computational cost can be used in a later step as a tiebreaker if there is any SSIM value collision.

[0029] Using the same perceived quality measure used previously in step 304, perceived quality is measured by comparing each of the re-rendered images with the original reference image, as shown in “Calculate Perceived Quality for Each Re-rendered Reference Image”, step 310. In the preferred embodiment, the structural similarity index (SSiM) is used to measure the perceived quality of the encoder results and will be used to measure the perceived quality of the new rendering results.

[0030] In the renderer, the two sets of perceived quality values, the SSiM values ​​for the encoded reference images calculated in step 304 and the SSiM values ​​for the profile-rendered reference images calculated in step 310, are compared across the two sets of images to find matching SSiM values ​​between the two sets. Ideally, for the SSiM value of each encoded reference image, there is an exact matching SSiM value from the profile-rendered image set. If there are no exact matches, the SSiM value of the chosen profile-rendered image should be higher and as close as possible to the SSiM value of the target encoded reference image.The corresponding SSiM values ​​in the two sets of perceived quality values ​​will identify a rendering quality settings profile for each QP value, as shown in “Finding a Quality Settings Profile for Each Partial Range of Encoder Settings,” step 312. In cases where there is one. Petition 870190104166, dated 10 / 15 / 2019, page 17 / 119 14 / 21 collision, in which there are two or more exact matches of the SSIM value set for the profile-reproduced images, the computational costs recorded in step 308 can be used as a tiebreaker and the least expensive rendering quality settings profile selected for the encoder configuration. FIG. 5 shows an example of a collision.

[0031] The encoder settings and their corresponding render quality settings profiles should be organized in a lookup table, as shown in “Creating a Lookup Table ASSIGNING A RENDER QUALITY SETTINGS PROFILE TO Each Encoder Setting,” step 314. This lookup table can be used at runtime in the renderer to change the render quality settings to match the quantization settings, as described in step 204 in FIG. 2. The lookup table provides a render quality settings profile that generates an image of the same perceived quality as the encoded frame and provides the greatest computational savings for the specified reference frame. Example lookup tables are shown in FIG. 4 and FIG. 5.

[0032] The lookup table generated by the method described in connection with FIG. 3 can be used in contexts, scenes, or game environments similar to those of the reference image. The process described in connection with FIG. 3 can be repeated for multiple reference images, each representing a specific environment, scene type, or other significant game context. For example, a reference image can be selected from each map in a game to generate multiple map-specific lookup tables. The lookup tables can also be combined to create a lookup table that can be used more generally in the game environment. For example, the map-specific lookup tables Petition 870190104166, dated 10 / 15 / 2019, page 18 / 119 Maps 15 / 21 can be combined to generate a lookup table that can be used for all maps in a game. To combine the lookup tables, the rendering quality settings profiles for each QP can be combined to find an average value for each setting contained in the profile. For example, three lookup tables are generated for three reference images. The rendering quality settings profiles are composed of three setting values: a post-processing quality setting, a shadow quality setting, and a resolution setting. To combine the rendering quality settings profiles for a QP value of 4, the profiles are read from each lookup table and represented as P41= {3, MEDIUM, 95%}, P42= {4, LOW, 90%}, and P4a= {2, MEDIUM, 90%}. The average values ​​are found for each setting to generate PAvg= {3, MEDIUM, 92%}.A profile averaging process should be rounded so that the rendering process never generates images with a perceived quality level lower than the current encoding quality setting. Profiles are calculated for each QP value and organized into a new lookup table.

[0033] FIG. 4 is an example of generating a lookup table to render quality configuration profiles that are composed of only one configuration. In this example, a single rendering quality configuration is adapted in response to changes in the encoder's quality settings. The rendering of a first-person view of a 3D scene is adapted in the renderer, changing the resolution of the 3D parts of the view, shown as “3D View” 400, while the resolution of the user interface (UI) elements, shown as “UI” 402, is not changed to maintain the readability of any text facing the player. This type of selective resolution scaling is called resolution scaling. Petition 870190104166, dated 10 / 15 / 2019, page 19 / 119 16 / 21 dynamic and is an increasingly common feature of rendering engines. The reference image, shown in “Reference Image” 404, represents a single frame from a typical video output rendered at the highest possible resolution and is chosen according to the guidelines described in step 300 of FIG. 3. In the encoder, the reference image, shown in “Reference Image” 404, is encoded for each integer QP value, as described in connection with step 302 of FIG. 3, to generate a list of encoded reference images in “Encoded Reference Images” 406. As described in connection with step 304 of FIG. 3, in the renderer, the SSIM values, shown as “SSIM” 408, are calculated for each encoded reference image 406.Since the rendering quality profile consists of only one quality setting, the number of quality profile permutations is limited to the number of possible values ​​available for the 3D view resolution, shown as “3D View” 400. The number of possible resolution values ​​is bounded by the maximum possible 3D view resolution and further bounded by the minimum feasible 3D view resolution. The aspect ratio can define how many resolution values ​​exist between the minimum and maximum resolutions. For example, a maximum resolution of 3840 x 2160 has an aspect ratio of 16:9, and the minimum feasible resolution at that aspect ratio is chosen as 1280 x 720. There are 160 possible resolutions with a 16:9 aspect ratio between these upper and lower limits. Alternatively, some number of the same resolutions between the upper and lower limits can be arbitrarily selected as resolution samples.For example, the resolution can be incrementally reduced in the x direction between 3840 and 1280 to select some number of sample resolution sizes.

[0034] In the renderer, the reference image is rendered again, as shown in “Reference Sequence Rendered Petition 870190104166, dated 10 / 15 / 2019, page 20 / 119 17 / 21 Again” 410, for each of the available resolution sizes or for each of the selected sample resolution sizes, as described in connection with step 308 of FIG. 3. The SSIM values ​​shown as “SSIM” 412 are calculated for each image re-rendered in the renderer, as described in step 310 of FIG. 3. The two sets of SSIM values, the SSIM values ​​for the encoded reference images, as shown in “SSIM” 408, and the SSIM values ​​for the profile-rendered reference images, as shown in “Re-rendered Reference Sequence” 410, are compared to locate matches in the image sets to provide a resolution setting for each integer QP value. The results are organized in a lookup table, as shown in “Lookup Table” 414, which will be used during runtime.By reducing the 3D view resolution to match quantization settings, wasted rendering work can be significantly reduced, which can result in additional benefits including reduced server power usage, reduced rendering times, and improved player feedback latency. These benefits compound in environments where multiple game instances are running on a single server.

[0035] FIG. 5 is an example of generating a lookup table for render quality configuration profiles containing multiple settings. The process as described in connection with FIG. 3 is unchanged for selecting a reference image and measuring the perceived quality for each encoder setting, as described in connection with steps 300, 302, and 304. Because the renderer can scale one or more render quality settings relative to the QP value, the list of generated render quality configuration profiles, described in connection with step 306 in FIG. 3, can Petition 870190104166, dated 10 / 15 / 2019, page 21 / 119 18 / 21 would be prohibitively long to facilitate re-rendering the reference image and calculating a perceived quality for each render quality settings profile. Since there can be a very large number of permutations of render settings, a decision tree can help to programmatically constrain the space of possibilities. For example, it might be undesirable to have a render quality settings profile where the post-processing quality is very low, but all other settings are very high. In certain modalities, it might be undesirable for high-quality shadows to be covered with low-quality post-processing. In other modalities, it might be the opposite.Decisions of this type are subjective, but based on criteria that include, but are not limited to, the computational cost associated with a specific rendering configuration, perceptual quality differences between two values ​​of a configuration, the comparative obviousness of one rendering configuration over another (such as enlarging effects that consume large portions of the screen compared to distant details that are only a few pixels wide), or the relative importance to gameplay (such as visual effects that are important for communicating feedback to the player).

[0036] FIG. 5 is an exemplary decision tree, as shown in “Decision Tree” 500, which is composed of one leaf for each permutation of four possible post-processing quality settings, three possible shadow quality settings, and five possible 3D view resolutions. This example decision tree is significantly smaller than a real-world example, as there may be many more adaptive rendering settings or many more options per setting, which will be apparent to those versed in the art. The decision tree is preferably traversed according to Petition 870190104166, dated 10 / 15 / 2019, page 22 / 119 19 / 21 any limiting conditions, such as avoiding sheets where the post-processing quality is very low but all other settings are high. For each sheet that is not removed by a limiting condition, the reference frame can be re-rendered with the rendering quality settings profile associated with the sheet, as described by 308 in FIG. 3. The computational cost, measured in rendering time or clock cycles, can be recorded at this point to be used as a potential tiebreaker in case of perceived quality value collision. Then, the perceived quality can be measured for each re-rendered image, as described in connection with step 310 of FIG. 3.For each perceived quality value (SSIM) calculated in the set calculated for the encoder settings, a list of all rendering quality setting profiles with a corresponding SSIM value can be generated as described in connection with step 312 of FIG. 3. The example in FIG. 5 shows this list being generated for a QP value of 16.

[0037] The SSIM value for the reference image encoded with QP value 16 is 0.997, for which there are three rendering quality settings profiles with corresponding SSIM values, shown with calculated computational costs of 16.004, 15.554, and 15.402. Since there are three collisions for the perceived quality value, the previously recorded computational costs serve as tiebreakers and can be used to determine which rendering quality settings profile is the cheapest, in this case, the one with a cost of 15.402. A lookup table, as shown in “Lookup Table” 502, must be generated to assign the cheapest rendering quality settings profile to each QP value, as described in step 314 in FIG. 3. The rendering quality settings profile selected for QP value 16 is shown in FIG. 5 as “PROFILE 16”. Petition 870190104166, dated 10 / 15 / 2019, page 23 / 119 20 / 21 EXAMPLE 1: Effects on Rendering Time as a Proxy for Computational Waste

[0038] In an example implementation, only the resolution is scaled linearly in response to changes in encoder quality. For example, if the encoder quality drops to 50%, the resolution will be reduced by 50% in response. Since rendering time savings correlate directly with computational power savings, rendering times were examined while the resolution was scaled. Measurements were taken in a low-motion environment, with a view comprising a first-person view of the player's hands, weapon, and a stationary wall. This low-motion view was selected to limit the number of factors that could contaminate the measurements, affecting the measured rendering times.These factors may include post-processing, such as motion blur, changes in the number of rendered objects, changes in on-screen textures, or other components of the view that are likely to change in high-motion views. Stationary viewing of a stationary scene also makes it possible to directly compare various measurements taken at scaled resolutions. The rendering engine was forced to output video at progressively lower resolutions, and the results were measured as shown in Table 1 below. Resolution Scale Pass Time Opaque Total Rendering Time 100% 0.4 ms 1.4 ms 50% 0.3 ms 1.0 ms 25% 0.2 ms 0.8 ms TABLE 1 Effects of Resolution Sizing Petition 870190104166, dated 10 / 15 / 2019, page 24 / 119 21 / 21 in Rendering Time

[0039] The opaque pass is the part of the rendering pipeline that draws opaque geometry in the viewport. This is the part of the rendering pipeline most sensitive to changes in resolution. Any rendering time savings or computational cost savings gained from scaling the resolution will primarily come from the opaque render pass.

[0040] As shown in Table 1, at a full resolution of 1280 x 720 in 60 frames, the rendering time for the opaque passthrough is 0.4 ms, for a total rendering time of 1.4 ms. When the resolution is reduced to 50% of the full resolution, the rendering time for the opaque passthrough is 0.3 ms, for a total rendering time of 1.0 ms. Scaling the resolution by 50% results in a significant rendering time saving of almost 30%. When the resolution is reduced to 25% of the full resolution, the rendering time for the opaque passthrough is 0.2 ms, for a total rendering time of 0.8 ms. Scaling the resolution by 75% results in a significant rendering time saving of over 40%.

[0041] The preceding description and drawings should be considered only as illustrative of the principles of the invention. The invention is not intended to be limited by the preferred embodiment and can be implemented in various ways that will be clear to those skilled in the art. Numerous applications of the invention will readily occur to those skilled in the art. Therefore, it is not desirable to limit the invention to the specific examples disclosed or to the exact construction and operation shown and described. Instead, all suitable modifications and equivalents may be used, encompassing the scope of the invention. Petition 870190104166, dated 10 / 15 / 2019, page 25 / 119

Claims

1 / 5 CLAIMS 1. A computer-implemented method for rendering, comprising the steps of: generating (306) one or more rendering quality setting profiles; generating (300) one or more reference images (404); encoding (302) the one or more reference images (406) for a partial range of encoder settings; calculating (304) a first perceived quality for each of the one or more reference images; re-rendering (308) the one or more reference images for each of the one or more rendering quality setting profiles; calculating (310) one or more second perceived qualities for each of the one or more re-rendered reference images (410);compare one or more first perceived qualities with one or more second perceived qualities, characterized in that a match between one or more first perceived qualities and one or more second perceived qualities results in an association of one or more encoder settings with their corresponding rendering quality settings profiles in one or more lookup tables; and generate a rendered image with substantially identical perceived quality to a frame encoded based on the one or more lookup tables (414).

2. A computer-implemented method according to claim 1, characterized in that it further comprises the step of calculating a computational cost for each of the reference images rendered again. Petition 870260050533, dated 05 / 27 / 2026, p. 5 / 21 2 / 5 3. A computer-implemented method according to claim 2, characterized in that it further comprises the step of applying the computational cost of each of the re-rendered reference images to the reference images and the re-rendered reference images to break multiple ties between one or more first-perceived qualities and one or more second-perceived qualities.

4. A computer-implemented method according to claim 2, characterized in that the computational cost is measured in rendering time or clock cycles.

5. A computer-implemented method according to claim 1, characterized in that an algorithmic image quality assessment is calculated and then used to calculate the first and second perceived qualities.

6. A computer-implemented method according to claim 1, characterized in that the association of one or more encoder configurations with their corresponding rendering quality configuration profiles is performed by applying a decision tree to programmatically reduce the possibility space.

7. A computer-implemented method according to claim 1, characterized in that a structural similarity index (SSIM) is used to measure first-perceived qualities and second-perceived qualities.

8. A computer-implemented method according to claim 1, characterized in that one or more lookup tables are generated using a plurality of integer values ​​or partial ranges of non-integer quantization parameters.

9. A computer-implemented method, according to Petition 870260050533, dated 05 / 27 / 2026, page 6 / 21 3 / 5, claim 8, characterized in that the integer values ​​of quantization parameters have a range from 0 to 51.

10. A computer-implemented method according to claim 1, characterized in that the quality settings profile is a list of values ​​for each available rendering quality setting.

11. A rendering system comprising: a codec that encodes one or more reference images to a partial range of encoder settings; and a renderer; wherein the renderer: generates one or more rendering quality setting profiles; generates the one or more reference images; calculates a first perceived quality for each of the one or more reference images; re-renders the one or more reference images for each of the one or more rendering quality setting profiles; calculates one or more second perceived qualities for each of the one or more re-rendered reference images;characterized by the fact that the renderer compares one or more first perceived qualities with one or more second perceived qualities, wherein a match between one or more first perceived qualities and one or more second perceived qualities results in an association of one or more encoder settings with their corresponding render quality settings profiles in one or more lookup tables; and generates a rendered image with perceived quality substantially identical to a frame encoded based on one or more lookup tables.

12. A rendering system according to claim 11, characterized in that it further comprises the step of calculating a computational cost for each of the reference images that are re-rendered.

13. A rendering system according to claim 12, characterized in that it further comprises the step of applying the computational cost of each of the re-rendered reference images to multiple match-offs between one or more first-perceived qualities and one or more second-perceived qualities, for reference images and re-rendered reference images.

14. A rendering system according to claim 12, characterized in that the computational cost is measured in rendering time or clock cycles.

15. A rendering system according to claim 11, characterized in that an algorithmic image quality assessment is calculated and then used to calculate the first and second perceived qualities.

16. A rendering system according to claim 11, characterized in that the association of one or more encoder configurations with their corresponding rendering quality configuration profiles is achieved by applying a decision tree to programmatically reduce the possibility space.

17. System for rendering, according to claim 11, characterized in that a structural similarity index (SSIM) is used to measure the first perceived qualities and the second perceived qualities.

18. A rendering system according to claim 11, characterized in that one or more lookup tables are generated using a plurality of integer values ​​or partial ranges of non-integer quantization parameters.

19. A rendering system according to claim 18, characterized in that the integer values ​​of quantization parameters have a range from 0 to 51.

20. A rendering system according to claim 11, characterized in that the quality settings profile is a list of values ​​for each available rendering quality setting. Petition 870260050533, dated 05 / 27 / 2026, p. 9 / 21

Images