Saving sample data in foveated rendering of graphics content
By rendering and sample point processing with different resolutions in the graphics content, the low rendering efficiency and artifact problems in VR and AR applications are solved, and efficient and low-latency graphics rendering effects are achieved.
Patent Information
- Application Number
- CN201980024756.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-01-15
- Filing Date
- 2019-03-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-03-29
AI Technical Summary
The prior art has problems with low rendering efficiency, high latency and high power consumption when rendering graphic content, especially in virtual reality (VR) and augmented reality (AR) applications, especially when it is necessary to render at the same time with high resolution and In low-resolution areas, the alias and artifacts introduced by Multiple Point Anti-aliasing (MSAA) are difficult to effectively solve.
By assigning the graphics content to multiple slices and rendering each area with different resolutions, rendering the user's fovea visual area with high resolution, rendering the surrounding area with low resolution, and downsampling of sample points when storing and displaying and save, using a variety of dot antialiasing technique to improve image quality, while saving sample points in low resolution areas to reduce information loss.
It improves rendering efficiency, reduces latency and power consumption, reduces the possibility of artifacts appearing, enhances user experience, and effectively solves the abstention and artifact problems introduced by MSAA.
Smart Images

Figure CN112041896B_ABST
Abstract
Description
[0001] This application claims the benefit of U.S. Provisional Patent Application No. 16 / 247,964, filed on January 15, 2019, which claims priority to Provisional Patent Application No. 62 / 657,415, filed on April 13, 2018, which is incorporated by reference in its entirety. Background Art
[0002] The electronic device may execute a program for displaying graphical content on a display. For example, the electronic device may execute a virtual reality (VR) program or an augmented reality (AR) program.
[0003] In some applications, graphics displayed on a display may be rendered at different resolutions in the same frame. For example, in a VR helmet, the center of the frame (e.g., an area intended for the user's foveal vision) may be rendered at a high resolution and the rest of the frame (e.g., an area intended for the user's peripheral vision) may be rendered at a low resolution. As an additional example, the electronic device may track the user's gaze (e.g., using a sensor) to identify the portion of the graphics that is in the user's foveal vision. If the electronic device determines that the user's eye movement follows a specific object displayed on the display, the electronic device may render the object at a high resolution and render other areas at a low resolution. Summary of the invention
[0004] Methods, devices, apparatus, and computer-readable media for generating graphical content are described herein. A method can include: assigning a first region of graphical content to a first of a plurality of slices, assigning a second region of graphical content to a second of the slices, determining a first sample set of the graphical content at the first slice and at a first resolution for each pixel of a plurality of pixels associated with the first region, determining a second sample set of the graphical content at the second slice and at a second resolution for each pixel of a plurality of pixels associated with the second region, downsampling the first sample set into a combined sample set, preserving samples of the second sample set to generate a third sample set having the preserved samples, storing the combined sample set, and storing the third sample set having the preserved samples.
[0005] The method can further include generating an image based on the stored combined sample point set and a third sample point set having the saved samples, and displaying the image on a virtual reality helmet or a wireless communication device.
[0006] In some embodiments, downsampling the first set of samples can include downsampling the first set of samples at a first scale, and preserving the samples can include downsampling the second set of samples by a second scale that is less than the first scale.
[0007] In further embodiments, saving the samples can include duplicating a second set of samples or upscaling a second plurality of samples in a second slice.
[0008] In further embodiments, boosting can include recent sampling or weighted sampling.
[0009] The method can further include determining that the first region is in the user's foveal vision, and based on the determination, determining to render the first region at a first resolution.
[0010] In some embodiments, the first area is predetermined to be rendered at a first resolution, and the first area is a central area of the graphics content.
[0011] The method can further include assigning a third region of the graphics content to a third tile of the plurality of tiles, and determining, at the third tile and at a third resolution lower than the first resolution, for each pixel of a plurality of pixels associated with the third region, a third set of samples of the graphics content, wherein the second resolution is different from the third resolution.
[0012] In some embodiments, the second resolution is at least one of half the first resolution, one quarter the first resolution, or one eighth the first resolution.
[0013] In yet further embodiments, the number of samples determined for each pixel of the graphical content is at least one of two, four, or eight.
[0014] In a further embodiment, storing the combined sample point set and storing the third sample point set with the saved samples includes storing the combined sample point set and the third sample point set with the saved samples in a system memory.
[0015] A device can include a processor and a memory coupled to the processor, the memory including instructions that, when executed by the processor, cause the device to perform a method, the method including: assigning a first region of graphics content to a first tile of a plurality of tiles, assigning a second region of graphics content to a second tile of the plurality of tiles, determining a first sample point set of the graphics content at the first tile and at a first resolution for each pixel of a plurality of pixels associated with the first region, determining a second sample point set of the graphics content at the second tile and at a second resolution lower than the first resolution for each pixel of a plurality of pixels associated with the second region, downsampling the first sample point set to a combined sample point set, saving samples of the second sample point set to generate a third sample point set having the saved samples, storing the combined sample point set, and storing the third sample point set having the saved samples.
[0016] A computer-readable medium can be a non-transitory computer-readable medium storing a program containing instructions that, when executed by a processor of a device, cause the device to perform a method, the method comprising: assigning a first area of graphics content to a first tile of a plurality of tiles, assigning a second area of graphics content to a second tile of the plurality of tiles, determining a first sample point set of the graphics content at the first tile and at a first resolution for each pixel of a plurality of pixels associated with the first area, determining a second sample point set of the graphics content at the second tile and at a second resolution lower than the first resolution for each pixel of a plurality of pixels associated with the second area, downsampling the first sample point set to a combined sample point set, saving samples of the second sample point set to generate a third sample point set having the saved samples, storing the combined sample point set, and storing the third sample point set having the saved samples.
[0017] An apparatus can include: means for assigning a first region of graphics content to a first slice of a plurality of slices, means for assigning a second region of graphics content to a second slice of the plurality of slices, means for determining a first set of samples of the graphics content at the first slice and at a first resolution for each pixel of a plurality of pixels associated with the first region, means for determining a second set of samples of the graphics content at the second slice and at a second resolution lower than the first resolution for each pixel of a plurality of pixels associated with the second region, means for downsampling the first set of samples to a combined set of samples, means for saving samples of the second set of samples to generate a third set of samples having the saved samples, means for storing the combined set of samples, and means for storing the third set of samples having the saved samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is an example diagram of a device capable of performing foveated rendering consistent with the disclosed embodiments.
[0019] Figure 2 is an example diagram of a device capable of performing foveated rendering consistent with the disclosed embodiments.
[0020] Figure 3 is an example illustration of determining samples at different resolutions in different slices consistent with the disclosed embodiments.
[0021] Figure 4A is an example illustration of downsampling in anti-aliasing at multiple points at different resolutions consistent with the disclosed embodiments.
[0022] Figure 4B is an example illustration of storing samples determined at different resolutions consistent with the disclosed embodiments.
[0023] Figure 5A is an example illustration of downsampling with preservation in multi-point anti-aliasing at different resolutions consistent with the disclosed embodiments.
[0024] Figure 5B is an example illustration of storing samples determined at different resolutions consistent with the disclosed embodiments.
[0025] Figure 6 is an example process for foveated rendering with diverse point anti-aliasing consistent with the disclosed embodiments.
[0026] Figure 7 is an example diagram of an electronic device capable of performing foveated rendering consistent with the disclosed embodiments. DETAILED DESCRIPTION
[0027] The systems, devices, computer-readable media, and methods disclosed herein can provide means for rendering regions of frames of computer graphics content differently based on the position of one or more eyes of a user relative to a coordinate system and / or a device (hereinafter referred to as "eye position"). For example, a high-resolution region and one or more low-resolution regions can be rendered in the same image / frame. As used herein, a high resolution can be any region higher than a low-resolution region. In various embodiments, a high resolution can be, for example, 1400X1200 pixels, 1080X1200 pixels, etc., and a low resolution can be, for example, half, one-quarter, one-eighth, one-sixteenth of a high resolution, etc. These resolutions are examples only and are not intended to be limiting. In some embodiments, a high resolution can represent the full resolution of a display device, and a low resolution can be a fractional resolution of the full resolution. In other embodiments, a high resolution can be higher or lower than the full resolution of a specific display device, and a high resolution can be downsampled or promoted before being displayed on a specific display device.
[0028] As used herein, a "region" of graphics content can refer to a regular grid in optical space corresponding to a subset of frames of the graphics content. In some embodiments, the regions can be the same for each frame of the graphics content. For example, each frame of the graphics content can be divided into the same 64 regions. In other embodiments, the number and shape of the regions can vary from frame to frame in the graphics content.
[0029] Rendering inefficiencies may be a current technical problem in computer graphics rendering. For example, generating graphical content using a VR program, application, and / or device may include rendering two views of the graphical content, which may render the content half as fast as before (which may cause some latency, limit available bandwidth on a hardware component bus, affect the frame rate of the display, affect motion to photon latency, etc.) and / or cause the device to use more power. One means of addressing the technical problem may be to render areas of the graphical content using parameters that utilize more processing resources (e.g., at a high resolution), and to render other areas of the scene using parameters that utilize fewer processing resources (e.g., at one or more low resolutions).
[0030] For example, in some embodiments, a user may focus primarily on the center of the graphical content. Therefore, portions away from the center (outer) are more likely to be in the user's peripheral vision, such that, for example, the user's visual system may not be able to recognize artifacts and / or appreciate details at the same level as in the center of the graphical content. Accordingly, the user experience may not be negatively impacted by rendering areas away from the center at one or more low resolutions, and rendering efficiency is increased by not rendering those areas at high resolutions.
[0031] In some instances, identifying the area where the user is staring can involve performing eye tracking to determine the area of the graphical content in the user's foveal vision. As used herein, the user's foveal vision can refer to the vision in the center of the user's field of view, where visual acuity is typically the highest. As an additional use herein, the area of graphical content in (or determined in) the user's foveal vision can refer to the area of such graphical content, which represents the intersection of the projection of the user's foveal vision and the two-dimensional cross-section of the graphical content that intersects the projection.
[0032] Once the area of graphical content in the user's foveal vision is determined, the means may include rendering the area at a high resolution and rendering the rest of the scene at one or more low resolutions. Again, because the area in the user's foveal vision remains high resolution, the user's experience may not be affected by rendering the area in the user's peripheral vision at one or more low resolutions, and rendering efficiency gains are made by not rendering those areas at a high resolution. As used herein, the user's peripheral vision can refer to vision outside the user's foveal vision.
[0033] As used herein, "foveated rendering" can refer to the process of rendering different areas of a scene based on different foveated parameters. Foveated parameters may include, but are not limited to, resolution, AA level, blur level, and filters to apply.
[0034] As used herein, a "gaze area" can refer to an area of one or more frames of graphics content rendered using parameters that utilize more processing resources. In some embodiments, a gaze area can be determined based on an area determined, predicted, and / or assumed to be within the user's foveal vision. For example, a gaze area can be an area rendered with high resolution, using a high AA level, a low blur level, a filter that improves regional quality, and the like. In contrast, a "non-gaze area" can refer to an area of one or more frames of graphics content rendered using parameters that utilize fewer processing resources. In some embodiments, a gaze area can be determined based on an area that is not determined, predicted, and / or assumed to be within the user's foveal vision (e.g., within the user's peripheral vision). For example, a non-gaze area can be an area rendered with low resolution, using a low AA level, a high blur level, a filter that improves processing efficiency, and the like.
[0035] For example, gaze areas can include rendering areas at a predetermined resolution (hereinafter “static foveated rendering”) and rendering areas at different resolutions based on eye tracking (hereinafter “dynamic foveated rendering”).
[0036] In some embodiments, foveated rendering can be used with multi-point anti-aliasing (MSAA). MSAA is a type of spatial anti-aliasing that can be used to improve image quality by supersampling multiple locations for pixels. MSAA is useful in many computer graphics use cases. For example, VR and AR helmets can place the display screen within a few inches of the user's eyes and the improved image quality is therefore obvious to the user. The examples used herein may refer to VR and AR, but various embodiments may not be limited to VR and AR, and mixed reality (MR) and augmented virtual (AV) and the like can also be applied. In summary, VR, AR, MR and AV may be referred to as extended display (XR). In various embodiments, all references to VR and AR herein may be applied alternatively or additionally to any other category of XR.
[0037] However, in some cases, MSAA may introduce aliasing and other artifacts because the texture is calculated only once for each pixel. Therefore, aliasing and other artifacts introduced by MSAA are another technical problem in computer graphics rendering.
[0038] As discussed in further detail below, these technical issues can be addressed by saving samples in low-resolution areas. In foveated rendering using MSAA, because there are multiple samples for at least some pixel locations that are ultimately displayed (e.g., in high-resolution areas), the samples are downsampled before being stored in system memory. However, samples from low-resolution areas include less information than high-resolution areas. Accordingly, if both types of areas are downsampled by the same ratio, the low-resolution area has less sample information than pixel locations. Additionally, when the low-resolution area is promoted after downsampling, the sample information will be lost. Therefore, foveated rendering using MSAA can be improved by saving samples of low-resolution areas.
[0039] As used herein, a "sample" or "sample value" can refer to a value representing the intersection of a channel (e.g., red, green, or blue) and a pixel. In some embodiments, a reference to a sample can include a sample having multiple sub-samples. For example, in a red, green, blue (RGB) color space, a sample can include three sub-samples, and in a red, green, blue, alpha (RGBA) color space, a sample can include four sub-samples.
[0040] As used herein, "saving" or "saved samples" may refer to a process performed on an area of one or more frames of graphics content (e.g., an area of low resolution), or the absence thereof, that results in more samples than by downsampling at a ratio that is equal to the ratio of downsampling performed on another area of the one or more frames of graphics content (e.g., an area of high resolution). For example, "saving" or "saved samples" may include downsampling a first area of one or more frames of graphics content at a ratio that is less than the ratio of downsampling performed on a second area of the one or more frames of graphics content. As an additional example, "saving" or "saved samples" may not include downsampling or upsampling and / or copying, not downsampling or upsampling a first area of one or more frames of graphics content, while downsampling is performed on a second area of the one or more frames of graphics content. As a further example, "saving" or "saved samples" can include upsampling a first area of one or more frames of graphics content, while downsampling is performed on a second area of the one or more frames of graphics content.
[0041] Various configurations are now described with reference to the accompanying drawings, wherein similar reference numerals may indicate functionally similar elements. The systems and methods generally described and illustrated in the drawings herein may be arranged and designed in a wide variety of different configurations. Therefore, as shown in the drawings, the following more detailed description of various configurations is not intended to limit the scope of the claims, but is merely a representative system method.
[0042] Figure 1An illustrative example of a device 100 capable of performing foveated rendering is depicted. As an illustrative example, the device 100 can include or be included in a graphics processing unit or any other type of processing unit (e.g., a central processing unit (CPU) or a digital signal processor (DSP)) that is configured to perform graphics processing operations using graphics content 102. In some examples, the graphics content 102 may correspond to graphics content from an AR or VR application.
[0043] The graphical content 102 can include one or more frames, such as the first frame 114. The first frame 114 can include a first region 108 and one or more second regions 109. In various embodiments, the graphical content 102 can be generated based on a state of an AR or VR application, based on location information of the device 100 (e.g., based on global positioning system (GPS) coordinates, local coordinates, locations of real-world objects, etc.), based on an orientation of the device 100, and / or based on a determined movement of a device, a user (e.g., head or eye position), or a real-world object captured by a camera of the device 100 (not shown).
[0044] In some embodiments, device 100 may include foveal estimation engine 104 (e.g., for dynamic foveated rendering). In other embodiments, foveal information may not be used, and device 100 may not include foveal estimation engine 104 (e.g., for static foveated rendering).
[0045] The device 100 can further include a rendering engine 112 that performs foveated rendering of the graphics content 102. For example, the first region 108 of the first frame 114 can be a region rendered at a high resolution. Accordingly, the first frame 114 in the rendering engine 112 can include a high-resolution region 116 based on the first region 108 of the first frame 114 of the graphics content 102. Additionally, the second region 109 of the first frame 114 can be a region rendered at a low resolution. For example, all second regions can be rendered at a quarter of the high resolution, one of the second regions can be rendered at a half of the high resolution and another second region can be rendered at an eighth of the high resolution, etc. The first frame 114 in the rendering engine 112 can include one or more low-resolution regions 118 that are based on one or more second regions 109 of the first frame 114 of the graphics content 102.
[0046] In some embodiments, rendering engine 112 may be, may include, or may be part of a GPU or another type of processor.
[0047] In an embodiment including a foveal estimation engine 104, the foveal estimation engine 104 can be configured to generate an indication 106 of a region of interest (e.g., a first region 108) of the graphical content 102. For example, the foveal estimation engine 104 can receive information about the eye position from a sensor (not shown) and determine a region of interest within a display within the user's foveal vision. In some embodiments, the region of interest can be associated with the first region 108 of the first frame 114. The foveal estimation engine 104 can send the indication of the region of interest to the rendering engine 112, and the rendering engine 112 can determine that the first region 108 should be a high-resolution region 116 and the second region 109 (not in the region of interest) should be one or more low-resolution regions 118.
[0048] In some embodiments, the fovea estimation engine 104 may be part of the same processor as the rendering engine 112, while in other implementations, the fovea estimation engine 104 may be part of a separate processor.
[0049] In some examples, for example, when the graphical content 102 and the device 100 correspond to an AR application, the device 100 may receive the scene information 110. For example, the scene information 110 may be captured by a camera (not shown) of the device 100, and the device 100 may overlay the graphical content 102 on the scene information 110.
[0050] Reference Figure 2 , depicts an illustrative example of a device 200 capable of performing foveated rendering. In various embodiments, the device 200 is capable of Figure 1 The device 100 is the same or similar to that in FIG. Figure 2 In the example of FIG. 2 , the device 200 can be configured to render graphics content using tile rendering and can include a tile engine 206 accordingly.
[0051] As described herein, tile rendering can refer to the process of dividing a frame of graphics content by a regular grid in optical space and rendering each portion of the grid (i.e., each tile) independently. Using tile rendering can provide technical advantages because bandwidth and storage can be immediately reduced compared to a rendering system that draws the entire frame. Accordingly, tile rendering can be useful in any class of devices that generate displays (e.g., laptops, desktop computers, advanced driver assistance systems, etc.), but is particularly common in mobile devices such as cellular phones (e.g., smartphones), VR and / or AR helmets, smart glasses, smart watches, etc.
[0052] As used herein, a "tile" can refer to a grid in optical space (e.g., a grid of 16x16 or 32x32 pixels) and / or a "tile" can refer to a portion of a tile memory in which sample values associated with a grid in optical space are stored.
[0053] In some embodiments, the tiling engine 206 may be a rendering engine (e.g., Figure 1 A part of the rendering engine 112 in FIG. Figure 2 As shown, the tiling engine 206 can receive the graphics content 201. In some embodiments, the tiling engine 206 can also receive an indication 205 of a region of interest (e.g., for dynamic foveated rendering) from the fovea estimation engine 203. In other embodiments, the region of interest may not be received by the tiling engine (e.g., for static foveated rendering).
[0054] The tiling engine 206 can be configured to perform tiling associated with processing of the graphics content 201. The tiling can include determining a set of tiles associated with the graphics content 201. For example, each frame of the graphics content 201 can be segmented into regions, where each region corresponds to a tile, multiple regions correspond to tiles, and / or a tile corresponds to multiple regions. For illustration, Figure 2 It is depicted that the first frame 207 can be segmented into a set of slices including a first slice 202 (eg, a first region of the first frame 207) and a second slice 204 (eg, a second region of the first frame 207). The first frame 207 can include additional slices for additional regions in the first frame 207.
[0055] exist Figure 2 In the example of , tiling engine 206 can be configured to render samples of graphics content 201 as a set of values for each tile associated with first frame 207 (e.g., by performing an operation (e.g., a vertex shading operation, a texturing operation, a rasterization operation, one or more other operations, or a combination thereof) to determine the value of one or more samples for each pixel in an area associated with the tile). In some embodiments, the sample value can be associated with a color value (e.g., an RGB color value) and can be determined based on the position of the sample in the frame of the graphics content and objects that overlap the position in the frame. As used herein, "pixel" can refer to a pixel in a final display (e.g., a 1920X1080 display has 2,073,600 pixels). Additionally, as used herein, "pixel" can refer to data in memory that ultimately corresponds to one pixel in the final display (if the resolution of the display is the same as in the memory), more than one pixel in the final display (if the resolution of the display is greater than in the memory), or less than one pixel and / or a portion of a pixel in the final display (if the resolution of the display is less than in the memory).
[0056] Each tile can be rendered independently, a value can be generated for each sample, and the value can be stored in the tile memory 208. If MSAA is used, multiple samples can be associated with each pixel position in the display, and a value can be generated for each sample. For example, the tile memory 208 can include values for the first tile 202, which can include multiple samples for the first pixel 212, multiple samples for the second pixel 214, multiple samples for the third pixel 216, and multiple samples for the fourth pixel 218. Additionally, the tile memory 208 can include values for the second tile 204, which can include multiple samples for the fifth pixel 222, multiple samples for the sixth pixel 224, multiple samples for the seventh pixel 226, and multiple samples for the eighth pixel 228.
[0057] In some embodiments, samples can be stored in system memory 230. For example, samples of first pixel 212 can be stored in system memory 230 as values for first pixel 232, samples of second pixel 214 can be stored in system memory 230 as values for second pixel 234, samples of seventh pixel 226 can be stored in system memory 230 as values for seventh pixel 236, and samples of eighth pixel 228 can be stored in system memory 230 as values for eighth pixel 238. Samples for third through sixth pixels can also be stored in system memory 230 as values.
[0058] In further embodiments, samples can additionally or alternatively be stored in other types of memory, such as, for example, local GPU memory (GMEM) or cache memory.
[0059] If MSAA is used, samples may be downsampled to generate a value (e.g., an RGB color value) for a pixel. For example, if 2XMSAA is used, two samples may be determined for each pixel in the display. For example, when the value is transferred to system memory 230, the samples may be downsampled (e.g., averaged together) to determine the value of the pixel in system memory 230. In some embodiments, as discussed in further detail below, samples from some slices (e.g., slices associated with a high resolution) may be downsampled, and other slices may have samples saved (e.g., slices associated with a low resolution). Additionally, as also discussed in further detail below, some slices (e.g., slices associated with a low resolution) may be upsampled to match the resolution of the display.
[0060] The value of the pixel stored in the system memory 230 can then be used to generate a display. For example, the value can indicate the RGB color value of the pixel, and the pixel can be set accordingly to generate the display.
[0061] Figure 2 The examples of illustrate aspects of tile rendering for generating graphical content. Consistent with the disclosed embodiments, other techniques may alternatively or additionally be used to generate graphical content.
[0062] Reference Figure 3 , shows an example illustration of determining samples at different resolutions in different slices. Figure 3 Frame 300 of graphical content is depicted. Frame 300 is shown for explanation purposes only and may not depict an actual frame that is displayed or stored. Frame 300 includes four regions, upper left, upper right, lower left, and lower right regions. The upper left region is rendered at a high resolution (compared to the other regions). For example, the upper left region of frame 300 may represent a region within the user's foveal vision, and the other regions may represent regions within the user's peripheral vision, as described above. Additionally, the other regions may be rendered at a quarter of the resolution of the upper left region (i.e., a resolution ratio of 4).
[0063] For explanation purposes, frame 300 depicts sample values that can be determined, where the sample values are associated with one of two colors (black or white). The upper left region rendered at a high resolution includes more samples than the upper right region, lower left region, and lower right region rendered at a low resolution.
[0064] When using tile rendering, frame 300 may not be rendered as an entire frame. Instead, frame 300 may be rendered in tiles (e.g., in a tile memory), where each region is associated with a tile, and each tile is rendered independently and sequentially. For example, the upper left region can be rendered in tile 310, the upper right region can be rendered in tile 312, the lower left region can be rendered in tile 314, and the lower right region can be rendered in tile 316. Accordingly, as shown in FIG. Figure 3 As depicted, the colors for the samples in the frame 300 correspond to the colors of the samples in the respective patches.
[0065] When using MSAA, the frame 300 and slices 310-316 may represent determined samples. For example, the samples may be determined with 2XMSAA or 4XMSAA. Accordingly, for 2XMSAA, two samples may be associated with one pixel in the system memory and / or display, and, for 4XMSAA, four samples may be associated with one pixel in the system memory and / or display. As described in further detail below, this may be the case for high-resolution areas, where the MSAA factor (e.g., two for 2XMSAA and four for 4XMSAA) represents the number of samples corresponding to one pixel in the system memory. Thus, in the high-resolution area, the samples may be downsampled at a ratio equal to the MSAA factor. However, in the low-resolution area, the MSAA factor may not represent the number of samples corresponding to one pixel in the system memory. As described in further detail below, the sample values in the low-resolution area may be saved (e.g., not downsampled or not downsampled at the same ratio as the high-resolution area) and stored in the system memory.
[0066] Reference Figure 4A , shows an example illustration of downsampling in multi-point anti-aliasing at different resolutions. Figure 4A Depicted are sheets 400, 402, 404, and 406, which in some embodiments may correspond to sheets 310, 312, 314, and 316, respectively (eg, Figure 3 4. As depicted in the drawings, slice 400 represents a slice rendered at a higher resolution (compared to the other slices). For example, slice 400 may represent an area within the user's foveal vision, and the other slices may represent areas within the user's peripheral vision, as described above. Additionally, slices 402-406 may be rendered at one-quarter the resolution of slice 400.
[0067] and Figure 3 Similarly, patches 400-406 depict sample values that can be determined, where the sample values are associated with one of two colors (black or white). Patch 400 rendered at a high resolution includes more samples than patches 402-406 rendered at a low resolution.
[0068] When using MSAA, some slices may be downsampled by the MSAA factor. Figure 4AIn the example depicted in , each slice is downsampled according to an MSAA factor of two (i.e., corresponding to 2XMSAA), and there is no difference in the sample values stored between the high-resolution slice (slice 400) and the low-resolution slices (slices 402-406). Accordingly, slice 400, which is a 4X4 slice, is downsampled to slice 410, which is a 2X4 slice, and slices 402, 404, and 406, which are 2X2 slices, are downsampled to slices 412, 414, and 416, which are 1X2 slices, respectively, and the combined slice values are averaged. As shown in FIG. Figure 4A As depicted, black patches combined with black patches are averaged into black patches, white patches combined with white patches are averaged into white patches, and black patches combined with white patches are averaged into gray patches.
[0069] Reference Figure 4B , shows an example illustration of storing samples at different resolutions. Figure 4B Depicted are sheets 420, 422, 424, and 426, which in some embodiments may be associated with sheets 410, 412, 414, and 416, respectively (eg, Figure 4A 4 (depicted). Slice 420 represents a slice rendered at a higher resolution (compared to the other slices) and downsampled so that some sample values are averaged (e.g., based on 2XMSAA, 4XMSAA, 8XMSAA, etc.). For example, slice 420 may represent an area within the user's foveal vision and the other slices may represent areas within the user's peripheral vision, as described above. Additionally, slices 422-426 may be rendered at one-quarter the resolution of slice 420 and may be downsampled (e.g., by the same factor as the higher resolution slices) so that some sample values are averaged.
[0070] Patches 420-426 depict sample values that may be determined, where the sample values are associated with one of three colors (black, white, and gray). Patches 420 rendered at a high resolution include more samples than patches 402-406 rendered at a low resolution.
[0071] Because tile 420 is rendered at a high resolution using MSAA and then downsampled, tile 420 may include a value for each pixel location in memory (e.g., system memory, GMEM, etc.) and / or on a display. Accordingly, the sample values in tile 420 may be copied to memory (shown in the upper left region of frame 430) and / or displayed on a display.
[0072] Because tiles 422-426 are rendered at a low resolution and then downsampled after using MSAA, the tiles may not include a value for each pixel location in system memory. Figure 4BAs depicted, slices 422-426 include one value for four pixel locations in system memory. Accordingly, slices 422-426 are promoted and then the sample values are stored. This means that, because one value represents four pixel locations, when slices 422, 424, and 426 are copied to the upper right region, lower left region, and lower right region of frame 430, respectively, one value is copied to each of the four pixel locations. Using one sample value to represent multiple pixel locations can be referred to as "nearest sampling" promotion because the nearest samples of multiple pixels are used as the values of the multiple pixels. Nearest sampling is just one of multiple possible promotion processes. For example, in other embodiments, promotion can be performed using "weighted sampling" promotion, where a value can be assigned to each pixel location based on a fractional weight of nearby samples, rather than copying sample values to multiple pixel locations. Example weighted sampling includes bilinear weighted sampling or bicubic weighted sampling.
[0073] In some embodiments, frame 430 may represent a frame rendered using foveated rendering and 2XMSAA, where low-resolution areas are rendered at one-quarter the resolution of high-resolution areas. Figure 3 The samples depicted may represent samples stored in system memory as frame 430. It is worth noting that because some sample values near the edges are averaged, there are differences between frame 300 and frame 430. Specifically, in the lower right region of frame 430, half of the pixel values are based on the average (rather than the original values), and this may result in visible artifacts when frame 430 is presented on a display.
[0074] Reference Figure 5A , shows an example illustration of utilizing preserved downsampling in multi-point anti-aliasing at different resolutions. Figure 5A Depicted are sheets 500, 502, 504, and 506, which in some embodiments may correspond to sheets 310, 312, 314, and 316, respectively (eg, Figure 3 500 represents a tile rendered at a higher resolution (compared to the other tiles). For example, tile 500 may represent an area within the user's foveal vision, and the other tiles may represent areas within the user's peripheral vision, as described above. Additionally, tiles 502-506 may be rendered at one-quarter the resolution of tile 500.
[0075] and Figure 3 Similarly, patches 500-506 depict sample values that can be determined, where the sample values are associated with one of two colors (black or white). Patch 500 rendered at a high resolution includes more samples than patches 502-506 rendered at a low resolution.
[0076] When using MSAA, some slices may be downsampled by the MSAA factor. Figure 5AIn the example depicted in , slice 500 is downsampled according to an MSAA factor of two (i.e., corresponding to 2XMSAA). However, samples in slices 502-506 may not be downsampled, but may be preserved. In this example, the samples may be preserved without downsampling because the resolution ratio of slices 502-506 (four in this example) is greater than or equal to the MSAA factor (two in this example). Accordingly, slice 500, which is a 4X4 slice, is downsampled to slice 510, which is a 2X4 slice, and slices 502, 504, and 506, which are 2X2 slices, are not downsampled and correspond to slices 512, 514, and 516, which are 2X2 slices, respectively, and the combined slice values are averaged. Accordingly, although in Figure 5A 502-506 are depicted differently from slices 512-516, which may represent the same values as slices 512-516. It is noteworthy that slice 516 having a saved sample value is identical to Figure 4A The slice 416 in which the sample value is not stored is different. Figure 5A As depicted, in patch 516 , the edge of the two colors is better preserved (compared to patch 416 ).
[0077] Reference Figure 5B , shows an example illustration of storing samples at different resolutions. Figure 5B Depicted are sheets 520, 522, 524, and 526, which in some embodiments may correspond to sheets 510, 512, 514, and 516, respectively (eg, Figure 5A ). Slice 520 represents a slice that is rendered at a high resolution and downsampled (compared to the other slices) so that some sample values are averaged (e.g., based on 2XMSAA, 4XMSAA, 8XMSAA, etc.). For example, slice 520 may represent an area within a user's foveal vision and the other slices may represent areas within the user's peripheral vision, as described above. Additionally, slices 522-526 may be rendered at one-quarter the resolution of slice 520 and the sample values preserved (in this example, without downsampling).
[0078] Patches 520-526 depict sample values that may be determined, where the sample values are associated with one of three colors (black, white, and gray). Patches 520 rendered at a high resolution include more samples than patches 502-506 rendered at a low resolution.
[0079] Because tile 520 is rendered at a high resolution using MSAA and then downsampled, tile 520 may include a value for each pixel location in memory (e.g., system memory, GMEM, a cache, etc.) Accordingly, the sample values in tile 520 may be copied to memory (shown in the upper left region of frame 530).
[0080] Because slices 522-526 are rendered at a low resolution, the slice may not include a value for each pixel location in system memory. However, because the sample values in the slice may be preserved (in this example, without downsampling), there may be more values for each pixel location than if downsampling had occurred. Figure 5B As depicted, slices 522-526 include one value for two pixel locations in system memory. Accordingly, slices 522-526 are promoted and then the sample values are stored. This means that, because one value represents two pixel locations, one value is copied to each of the two pixel locations when slices 522, 524, and 526 are copied to the upper right region, lower left region, and lower right region of frame 530, respectively.
[0081] In some embodiments, frame 530 may represent a frame rendered using foveated rendering and 2XMSAA, where low-resolution areas are rendered at one-quarter the resolution of high-resolution areas. Figure 3 The samples depicted may represent samples stored in system memory as frame 530. It is noteworthy that there are differences between frame 300 and frame 530 because some sample values near the edge are averaged. However, the lower right area of frame 530, where the samples are stored, includes more Figure 4B Frame 430 in which no samples are stored has less averaged area (in this case, no averaged area). Therefore, when frame 530 is displayed on a display, visible artifacts may be reduced compared to when frame 430 is displayed on a display.
[0082] Reference Figure 6 , shows an example process 600 for foveated rendering with multi-point anti-aliasing. In various embodiments, the example process 600 can be performed by a computing device (e.g. Figure 1 The depicted device 100, Figure 2 9 ). Additionally or alternatively, in some embodiments, the computing device can be a VR helmet, an AR helmet, a cellular phone (or other wireless communication device), a laptop computer, a tablet computer, a desktop computer, a server, a group of network devices, an advanced driver assistance system, a smart watch, smart glasses, etc. In various embodiments, the computing device can include a display device, can be connected to a display device, and / or can send information to a display device.
[0083] In box 610, the computing device can divide the graphics content into regions. In some embodiments, the regions can correspond to the coordinates of the image for display and the regions can be of the same size. In other embodiments, the region size can vary. In some implementations, the region can be static and / or predetermined. In other words, for the computing device and / or the application that renders the graphics content, the region is pre-set, and the computing device can use the same region in each instance of rendering the graphics content or in each instance of rendering the graphics content for the application. In other implementations, the region can be dynamic and derived by the computing device based on, for example, the complexity of the graphics content, the available memory, the priority of the running application, etc. In other words, when the application is generating the graphics content and / or in different instances of the application, the region can vary.
[0084] In block 620, the computing device can determine a resolution for each region. In some embodiments, the computing device can determine to associate one or more regions with a high resolution and to associate one or more regions with a low resolution (e.g., half, one quarter, etc. of the high resolution). In other embodiments, the computing device can associate one or more regions with a high resolution, associate one or more regions with a first low resolution (e.g., half of the high resolution), associate one or more regions with a second low resolution (e.g., one quarter of the high resolution), etc. Accordingly, there can be any number of different low-resolution regions.
[0085] In a static foveated rendering embodiment, the resolution of each region can be predetermined. For example, the computing device can associate one or more central regions with a high resolution, and one or more regions away from the center with one or more low resolutions.
[0086] In a dynamic foveated rendering embodiment, the resolution of each region can be determined based on eye tracking information. For example, based on the eye tracking information, one or more regions in the user's foveal vision can be identified, and these one or more regions can be associated with high-resolution regions, and the remaining regions can be associated with low resolution. As a further example, one or more regions in the user's foveal vision can be associated with high resolution, one or more regions closest to the high-resolution region can be associated with a first low resolution (e.g., one quarter of the high resolution), and the remaining one or more regions can be associated with a second low resolution (e.g., one eighth of the high resolution).
[0087] In block 630, regions can be assigned to tiles (eg, in a tile memory). In some embodiments, each region can be assigned to a tile, while in other embodiments, multiple regions can be assigned to a tile or a region can be split and assigned to multiple tiles.
[0088] In block 640, the computing device can determine samples of the graphics content. The samples can be determined at a resolution associated with the region (e.g., a high resolution, a first low resolution, a second low resolution, etc.). Additionally, the samples can be determined using MSAA (e.g., 2XMSAA, 4XMSAA, 8XMSAA, 16XMSAA, etc.). Accordingly, multiple samples can be determined for each pixel associated with the region. In some embodiments, the regions (split into slices) can be processed sequentially to reduce overall processing, memory, and bandwidth used. In further embodiments, the regions can be processed in parallel to reduce processing time.
[0089] In block 650 , the computing device can store the samples in the slice into memory (eg, into system memory, GMEM, cache, etc.).
[0090] In block 652, for the high-resolution region, the computing device can downsample the samples. For example, if 2XMSAA is used, for each pixel position in the high-resolution region, the computing device can combine and average two samples, and if 4XMSAA is used, for each pixel position in the high-resolution region, the computing device can combine and average four samples. The values of the downsampled samples can be stored in a memory.
[0091] In block 654, for a low-resolution region where the resolution ratio is less than the MSAA factor, the computing device can save and downsample the samples. In some embodiments, the computing device can save and downsample the samples by dividing the MSAA factor by the resolution ratio instead of the MSAA factor. For example, if the resolution ratio of the low-resolution region is two (half the resolution of the high resolution) and the MSAA factor is four (4XMSAA), the computing device can downsample the region by two. The values of the downsampled samples can be stored in memory. Accordingly, some level of detail in the samples is saved compared to downsampling all regions by the MSAA factor.
[0092] Some embodiments may not include areas of low resolution where the resolution ratio is less than the MSAA factor, and therefore, the computing device may not perform block 654 .
[0093] In block 656, for the low-resolution area with a resolution ratio equal to the MSAA factor, the computing device can save the samples. In some embodiments, the computing device can save by not downsampling and / or bypassing downsampling of the samples. The sample values in the slice memory can be copied and stored in the memory. Accordingly, the samples are saved.
[0094] Some embodiments may not include areas of low resolution where the resolution ratio is equal to the MSAA factor, and therefore, the computing device may not perform block 656 .
[0095] In block 658, for low-resolution areas where the resolution ratio is greater than the MSAA factor, the computing device can preserve and upsample (e.g., as described with respect to Figure 4A and 4B ). In some embodiments, the computing device can save by not downsampling and / or bypassing downsampling of samples. Additionally, because there are fewer samples than pixel locations, the samples can be boosted by using sample values for multiple pixel locations (nearest sampling) in some examples, or using fractional portions of nearby samples (weighted sampling) in other examples. Accordingly, because downsampling is not performed, samples are saved.
[0096] Some embodiments may not include areas of low resolution where the resolution ratio is greater than the MSAA factor, and therefore, the computing device may not perform block 658 .
[0097] At block 660, the computing device can generate an image for display on a display device based on the samples stored in the memory. For example, the computing device can use the sample values to adjust pixel values in an image file and / or on a display device.
[0098] refer to Figure 7 , depicts an example diagram of an electronic device generally designated as electronic device 700 and capable of performing foveated rendering. The electronic device 700 may correspond to a mobile device (e.g., a wireless communication device such as a cellular phone), a computer (e.g., a server, a laptop, a tablet, or a desktop computer), an access point, a base station, a wearable electronic device (e.g., a personal camera, a head mounted display, a VR helmet, an AR helmet, smart glasses, or a smart watch), a vehicle control system or console (e.g., an advanced driver assistance system), an autonomous vehicle (e.g., a robot car or a drone), a home appliance, a set-top box, an entertainment device, a navigation device, a personal digital assistant (PDA), a television, a monitor, a tuner, a radio (e.g., a satellite radio), a music player (e.g., a digital music player or a portable music player), a video player (e.g., a digital video player, such as a digital video disc (DVD) player or a portable digital video player), a robot, a healthcare device, other electronic devices, or combinations thereof.
[0099] The electronic device 700 includes one or more processors, such as a processor 710 and a graphics processing unit (GPU) 796. The processor 710 may include a central processing unit (CPU), a digital signal processor (DSP), another processing device, or a combination thereof. Figure 7In the example of FIG. 7 , the processor 710 may include the fovea estimation engine 704, and the GPU 796 may include the rendering engine 712. In other embodiments, the rendering engine 712 may be included in the processor 710 and / or the fovea estimation engine 704 may be included in the GPU 796, etc.
[0100] The processor 710 may be coupled to the GPU 796. In an illustrative example, the processor 710 may be configured to communicate with the GPU 796 using graphics application programming interface (API) calls. For example, the GPU 796 may include an API call analyzer (not shown), and the processor 710 may be configured to provide the API calls to the API call analyzer during graphics processing performed by the GPU 796.
[0101] The electronic device 700 may further include one or more memories, such as memory 732. The memory 732 may be coupled to the processor 710, the GPU 796, or both. The memory 732 may include a random access memory (RAM), a magnetoresistive random access memory (MRAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), one or more registers, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), another storage device, or a combination thereof.
[0102] Memory 732 may store instructions 760. Instructions 760 may be executable by processor 710, GPU 796, or both. In some embodiments, instructions 760 may instruct processor 710 and / or GPU 796 to execute Figure 6 The process described.
[0103] A coder / decoder (CODEC) 734 can also be coupled to the processor 710. The CODEC 734 can be coupled to one or more microphones, such as microphone 738. The CODEC 734 can also be coupled to one or more speakers, such as speaker 736. The CODEC 734 can include a memory 790 storing instructions 795 executable by the CODEC 734.
[0104] Figure 7 Also shown is a display controller 726 coupled to the controller 710 and the display 728. In some embodiments, the electronic device 700 may further include a wireless controller 740 coupled to an antenna 742. In other embodiments, the electronic device 700 may further include a wired network controller (not shown) coupled to a computer network cable.
[0105] The electronic device 700 may further include a sensor 750 (e.g., a camera). The sensor 750 may be any type of sensor configured to capture images of the user's eyes. The fovea estimation engine 704 may be configured to identify eye position information based on the captured images from the sensor 750.
[0106] In a specific example, the processor 710, GPU 796, memory 732, display controller 726, codec 734, and wireless controller 740 are included in a system on chip (SoCial) device. In addition, an input device 730 and a power supply 744 can be coupled to the SoC device 722. In addition, in a specific example, as Figure 7 As shown, the display 728, input device 730, speaker 736, microphone 738, antenna 742, power supply 744, and sensor 750 are external to the SoC device 722. However, each of the display 728, input device 730, speaker 736, microphone 738, antenna 742, power supply 744, and sensor 750 can be coupled to a component of the SoC device 722, such as an interface or controller.
[0107] As used herein, "coupling" may include communication coupling, electrical coupling, magnetic coupling, physical coupling, optical coupling, and combinations thereof. Two devices (or components) may be coupled (e.g., communication coupling, electrical coupling, or physical coupling) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. The same or different devices may include two electrically coupled devices (or components), and may be connected via electronics, one or more connectors, or inductive coupling as described in the non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled (such as electronic communications) may directly or indirectly send and receive electronic signals (digital signals or virtual signals) via, for example, one or more wires, buses, networks, etc.
[0108] The aforementioned disclosed devices and functions may be designed and represented using computer files (e.g., computer files in file formats including Graphic Database System II (GDSII), GERBER, etc.). The computer files may be stored on a computer readable medium. Some or all of such files may be provided to a fabrication handler that fabricates devices based on such files. The resulting products include wafers that are subsequently cut into dies and packaged into integrated circuits (or "chips"). These are then processed in a computer program such as a processor. Figure 7 The electronic device 700 is an electronic device using an integrated circuit.
[0109] Although some examples are described separately for convenience, it should be noted that various aspects of such examples can be appropriately combined without departing from the scope of the disclosure. For example, the device 100 can be configured to be based on the reference Figure 2 , 3 , 4A, 4B, 5A and 5B. Those skilled in the art will recognize other such modifications within the scope of the present disclosure.
[0110] The various illustrative logic blocks, configurations, modules, circuits, and algorithm steps described together with the examples disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. According to the functions of the various illustrative components, blocks, configurations, modules, circuits, and steps, they are generally described above. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the impact of design constraints on the overall system. For specific applications, technicians can implement the functions in a varying manner, but such implementation decisions should not be interpreted as causing deviations from the scope of the present disclosure.
[0111] One or more operations of the methods or algorithms described herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. For example, it may be initiated, controlled, or executed by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a CPU, a digital signal processor (DSP), a controller, another hardware device, a firmware device, or a combination thereof. Figure 6 One or more operations of process 600 of the present invention. The software module may reside in RAM, MRAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of non-temporary storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Additionally or alternatively, the storage medium may be indispensable to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a computing device or a user terminal. Additionally or alternatively, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.
[0112] The foregoing description of examples of the present disclosure is provided to enable those skilled in the art to make or use the disclosed examples. Various modifications to these examples will be apparent to those skilled in the art, and the principles defined herein may be applied to other examples without departing from the scope of the present disclosure. Therefore, the disclosure presented is not intended to be limited to the examples shown herein, but rather to the widest possible scope consistent with the principles and novel features defined by the following claims.
Claims
1. A device for generating graphic content, comprising: Memory; as well as A processor coupled to the memory, configured to cause the device to perform a method, comprising: assigning a first region of the graphical content to a first slice of the plurality of slices, the first region corresponding to the graphical content in the user's foveal vision; assigning a second region of the graphical content to a second slice of the plurality of slices, the second region corresponding to graphical content in peripheral vision outside of the foveal vision; determining, at the first patch and at a first resolution, for each pixel of a plurality of pixels associated with the first region, a first plurality of samples of the graphics content; determining, at the second patch and at a second resolution lower than the first resolution, for each pixel of a plurality of pixels associated with the second region, a second plurality of samples of the graphics content; downsampling the first plurality of samples into a combined plurality of samples; saving samples of the second plurality of samples to generate a third plurality of samples having the saved samples; storing the combined plurality of samples; and The third plurality of samples having the saved samples is stored.
2. The device according to claim 1, wherein the device comprises a virtual reality helmet or a wireless communication device, and the method further comprises: generating an image based on the stored combined plurality of samples and the third plurality of samples having the saved samples; as well as The image is displayed.
3. The apparatus according to claim 1, wherein: Downsampling the first plurality of samples comprises downsampling the first plurality of samples at a first ratio; and Saving the samples includes downsampling the second plurality of samples at a second ratio that is smaller than the first ratio.
4. The apparatus of claim 1, wherein saving the sample points comprises replicating the second plurality of sample points in the second slice. The apparatus of claim 1 , wherein preserving the sample points comprises promoting the second plurality of sample points. The apparatus of claim 5 , wherein the boosting comprises a most recent sample. The apparatus of claim 5 , wherein the boosting comprises weighted sampling.
8. The apparatus of claim 1, the method further comprising: determining that the first region is in the user's foveal vision; as well as Based on determining that the first region is in the user's foveal vision, determining to render the first region at the first resolution.
9. The apparatus of claim 1, wherein: Predetermining to render the first area at the first resolution; and The first area is a central area in the graphic content.
10. The apparatus of claim 1, the method further comprising: assigning a third region of the graphical content to a third slice of the plurality of slices; as well as A third plurality of samples of the graphics content is determined for each pixel of a plurality of pixels associated with the third region at the third patch and at a third resolution lower than the first resolution, wherein the second resolution is different than the third resolution.
11. The apparatus of claim 1, wherein the second resolution is at least one selected from the group consisting of a half of the first resolution, a quarter of the first resolution, and an eighth of the first resolution.
12. The apparatus of claim 1, wherein the number of samples determined for each pixel of the graphic content is at least one selected from the group consisting of two, four, and eight.
13. The apparatus of claim 1, wherein storing the combined plurality of samples and storing the third plurality of samples with saved samples comprises storing the combined plurality of samples and the third plurality of samples with saved samples in a system memory.
14. A method for generating graphical content comprising: assigning a first region of graphical content to a first slice of the plurality of slices, the first region corresponding to the graphical content in the user's foveal vision; assigning a second region of the graphical content to a second slice of the plurality of slices, the second region corresponding to graphical content in peripheral vision outside of the foveal vision; determining, at the first patch and at a first resolution, for each pixel of a plurality of pixels associated with the first region, a first plurality of samples of the graphics content; determining, at the second patch and at a second resolution lower than the first resolution, for each pixel of a plurality of pixels associated with the second region, a second plurality of samples of the graphics content; downsampling the first plurality of samples into a combined plurality of samples; saving samples of the second plurality of samples to generate a third plurality of samples having the saved samples; storing a plurality of sample points of the combination; as well as The third plurality of samples having the saved samples is stored.
15. The method according to claim 14, further comprising: generating an image based on the stored combined plurality of samples and the third plurality of samples having the saved samples; as well as The image is displayed on a virtual reality helmet or a wireless communication device.
16. The method of claim 14, wherein: Downsampling the first plurality of samples comprises downsampling the first plurality of samples at a first ratio; and Saving the samples includes downsampling the second plurality of samples at a second ratio that is smaller than the first ratio.
17. The method of claim 14, wherein saving the sample points comprises replicating the second plurality of sample points in the second slice.
18. The method of claim 14, wherein: Saving the sample points includes promoting the second plurality of sample points; and The boosting includes recent sampling or weighted sampling.
19. The method of claim 14, further comprising: determining that the first region is in the user's foveal vision; as well as Based on determining that the first region is in the user's foveal vision, determining to render the first region at the first resolution.
20. A non-transitory computer-readable medium for generating graphical content, the non-transitory computer-readable medium storing a program including instructions that, when executed by a processor of a device, cause the device to perform a method comprising: assigning a first region of graphical content to a first slice of the plurality of slices, the first region corresponding to the graphical content in the user's foveal vision; assigning a second region of the graphical content to a second slice of the plurality of slices, the second region corresponding to graphical content in peripheral vision outside of the foveal vision; determining, at the first patch and at a first resolution, for each pixel of a plurality of pixels associated with the first region, a first plurality of samples of the graphics content; determining, at the second patch and at a second resolution lower than the first resolution, for each pixel of a plurality of pixels associated with the second region, a second plurality of samples of the graphics content; downsampling the first plurality of samples into a combined plurality of samples; saving samples of the second plurality of samples to generate a third plurality of samples having the saved samples; storing a plurality of sample points of the combination; as well as The third plurality of samples having the saved samples is stored.
