Content-Based Foveation

Abstract

Before rendering a current frame, the device retrieves a previously rendered frame. The device determines that a first portion of the previously rendered frame is associated with a particular type of content. The device renders a first portion of the current frame that corresponds to the first portion of the previously rendered frame with first rendering characteristics, and renders a second portion of the current frame with second rendering characteristics that are different from the first rendering characteristics.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 455,599, filed March 30, 2023, which is incorporated by reference in its entirety.

[0002] This disclosure relates generally to content-based foveation. [Background technology]

[0003] Some devices include a display that displays visual content. Presenting the visual content may include rendering the visual content using a renderer. Displaying the visual content at a relatively high resolution tends to enhance the user experience provided by the device. However, rendering the visual content at a relatively high resolution may be resource intensive, thereby draining the device's battery.

[0004] The present disclosure may have a more detailed description, as can be understood by those skilled in the art, by reference to aspects of several exemplary implementations, some of which are illustrated in the accompanying drawings. [Brief explanation of the drawings]

[0005] [Figure 1A] FIG. 1 illustrates an exemplary operating environment according to some implementations. [Figure 1B] FIG. 1 illustrates an exemplary operating environment according to some implementations. [Figure 1C] FIG. 1 illustrates an exemplary operating environment according to some implementations.

[0006] [Figure 2] FIG. 1 is a diagram of a content presentation system according to some embodiments.

[0007] [Figure 3] 1 is a flowchart representation of a content-based frame foveating method according to some implementations.

[0008] [Figure 4] FIG. 1 is a block diagram of a device for foveating content-based frames according to some implementations.

[0009] According to common practice, the various features illustrated in the figures may not be drawn to scale. Accordingly, dimensions of various features may be arbitrarily increased or decreased for clarity. In addition, some drawings may not depict all of the components of a given system, method, or device. Finally, like reference numerals may be used throughout the specification and figures to indicate like features. Summary of the Invention

[0010] Various implementations disclosed herein include devices, systems, and methods for foveating frames based on content. In some implementations, the method is performed by a device including a display, non-transitory memory, and one or more processors coupled to the display and the non-transitory memory. In various implementations, the method includes obtaining a previously rendered frame before rendering a current frame. In some implementations, the method includes determining that a first portion of the previously rendered frame is associated with a particular type of content. In some implementations, the method includes rendering a first portion of the current frame corresponding to the first portion of the previously rendered frame using first rendering characteristics and rendering a second portion of the current frame using second rendering characteristics that are different from the first rendering characteristics.

[0011] According to some implementations, a device includes one or more processors, a plurality of sensors, a non-transitory memory, and one or more programs. In some implementations, the one or more programs are stored in the non-transitory memory and executed by the one or more processors. In some implementations, the one or more programs include instructions for performing or causing to be performed any of the methods described herein. According to some implementations, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by one or more processors of the device, cause the device to perform or cause to be performed any of the methods described herein. According to some implementations, a device includes one or more processors, a non-transitory memory, and means for performing or causing to be performed any of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] Numerous details have been described to provide a thorough understanding of the exemplary implementations shown in the drawings. However, the drawings merely illustrate some exemplary aspects of the present disclosure and therefore should not be considered limiting. Those skilled in the art will understand that other useful aspects and / or variations do not include all of the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the exemplary implementations described herein.

[0013] A device that displays visual content may include a renderer that renders frames that may be displayed on the device's display. The renderer may be capable of rendering frames with various rendering characteristics. For example, the renderer may be capable of rendering frames at a relatively high resolution and a relatively low resolution. Rendering frames at a relatively high resolution may be resource intensive. Also, because some types of content can be rendered at a relatively low resolution without adversely affecting viewability, rendering such types of content at a high resolution tends to unnecessarily utilize computing resources.

[0014] The present disclosure provides methods, systems, and / or devices for performing content-based foveation. Performing content-based foveation allows a device to render different portions of a current frame using different rendering characteristics. Utilizing rendering characteristics that require fewer resources to generate at least some portions of the current frame tends to conserve resources. During operation, the device determines that a first portion of the current frame will be rendered using a previously rendered frame with first rendering characteristics and that a second portion of the current frame will be rendered using a second rendering characteristic that is less resource-intensive than the first rendering characteristics. For example, the device estimates that a first portion of the current frame will be rendered using a previously rendered frame with a relatively high resolution and that a second portion of the current frame will be rendered using a relatively low resolution. In this example, the device conserves resources by not rendering the second portion with the relatively high resolution.

[0015] Before rendering the current frame, the device determines which portions of the current frame are likely to be associated with infrequent content and which portions of the current frame are likely to be associated with frequent content. The frequent content will be rendered at a relatively high resolution to provide an acceptable level of viewability, while the infrequent content can be rendered at a relatively low resolution to conserve rendering resources. Rendering the infrequent content at a relatively low resolution tends to conserve computing resources and extend the device's battery life.

[0016] Because the device cannot perform content analysis on the current frame before rendering it, the device can utilize previously rendered frames to estimate which portions of the current frame are likely to represent high-frequency content and which portions of the current frame are likely to represent low-frequency content. The previously rendered frames may be low-resolution renderings of the current frame. Alternatively, the previously rendered frames may correspond to an earlier time. The device can reproject the previously rendered frames based on head pose data to reduce differences between the previously rendered frames and the current frame being rendered.

[0017] The device may identify low-frequency and high-frequency portions of the previously rendered frame based on the respective compression efficiencies of various portions of the previously rendered frame. The low-frequency portions of the previously rendered frame may have higher compression efficiency because the low-frequency portions can be compressed to a greater extent due to less variance in texture values. The high-frequency portions of the previously rendered frame tend to have lower compression efficiency because the high-frequency portions can be compressed to a lesser extent due to greater variance in texture values.

[0018] The device may utilize separate renderers to render the low-frequency and high-frequency portions of the current frame. For example, the device may utilize a low-frequency renderer to render the low-frequency portion of the current frame and a high-frequency renderer to render the high-frequency portion of the current frame. Additionally or alternatively, the device may utilize shaders of different complexity to render the low-frequency and high-frequency portions of the current frame. For example, the device may use a relatively low-complexity shader to render the low-frequency portion of the current frame and a relatively high-complexity shader to render the high-frequency portion of the current frame.

[0019] FIG. 1A is a diagram illustrating an exemplary physical environment 10 according to some implementations. While relevant features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for brevity's sake so as not to obscure more pertinent aspects of the exemplary implementations disclosed herein. Thus, by way of non-limiting example, the physical environment 10 includes an electronic device 20 and a user 12 of the electronic device 20. In some implementations, the electronic device 20 includes a handheld computing device that may be held by the user 12. For example, the electronic device 20 may include a smartphone, tablet, media player, laptop, etc. that may be held by the user 12. Alternatively, in some implementations, the electronic device 20 includes a wearable computing device that may be worn by the user 12. For example, the electronic device 20 may include a head-mounted device (HMD) or an electronic watch that may be worn by the user 12.

[0020] In various embodiments, electronic device 20 includes a display 22 and a content presentation system 200. Content presentation system 200 generates visual content, and display 22 displays the visual content. In the example of Figure 1A, display 22 is displaying a previously rendered frame 30 that was rendered by content presentation system 200. Previously rendered frame 30 shows a building 32, a courtyard 34, a first tree 36, a second tree 38, a lawn 40 around building 32, and clouds 42.

[0021] In various implementations, the content presentation system 200 renders the current frame based on the type of content represented by the previously rendered frames 30. Referring to FIG. 1B , in various implementations, the content presentation system 200 identifies the type of content represented by each portion of the previously rendered frames 30. In the example of FIG. 1B , the content presentation system 200 determines that a first portion 50 of the previously rendered frames 30 represents a particular type of content. The content presentation system 200 may determine that a second portion 70 of the previously rendered frames 30 (e.g., the remainder of the previously rendered frames 30) does not represent the particular type of content.

[0022] In some implementations, the content presentation system 200 performs a frequency analysis on the previously rendered frames 30 to identify a particular type of content represented by a first portion 50 of the previously rendered frames 30. In some implementations, the content presentation system 200 generates a discrete cosine transform (DCT) for the previously rendered frames 30. The DCT of the previously rendered frames 30 may indicate that the first portion 50 of the previously rendered frames 30 is associated with a first frequency 52 that is higher than a threshold frequency 60. For example, the DCT of the previously rendered frames 30 may indicate that the first portion 50 of the previously rendered frames 30 represents high-frequency content. The DCT of the previously rendered frames 30 may indicate that a second portion 70 of the previously rendered frames 30 (e.g., the remainder of the previously rendered frames 30) is associated with a second frequency 72 that is lower than the threshold frequency 60. For example, the DCT of the previously rendered frames 30 may indicate that the second portion 70 of the previously rendered frames 30 represents low-frequency content.

[0023] In some implementations, the content presentation system 200 performs a reprojection operation on the previously rendered frame 30 before determining which portions of the previously rendered frame 30 include high-frequency content. The content presentation system 200 reprojects the previously rendered frame 30 based on head pose data indicative of head movement of the user 12 so that the previously rendered frame 30 more closely represents the current frame that the content presentation system 200 must render. Reprojecting the previously rendered frame 30 based on the head pose data results in the previously rendered frame 30 being updated to reflect head movement of the user 12 since the previously rendered frame 30 was rendered. Reprojecting the previously rendered frame 30 may include warping the previously rendered frame 30 based on the head pose data.

[0024] 1C , in various implementations, the content presentation system 200 renders the current frame 130 based on the type of content represented by the previously rendered frame 30 shown in FIGS. 1A and 1B. In the example of FIG. 1C , the content presentation system 200 renders the first portion 150 of the current frame 130, which corresponds to the first portion 50 of the previously rendered frame 30 (shown in FIG. 1B), using a first rendering characteristic 152. The first rendering characteristic 152 is greater than a threshold rendering characteristic 160. In some implementations, the first rendering characteristic 152 is a first resolution, and the threshold rendering characteristic 160 is a threshold resolution. In some implementations, the first rendering characteristic 152 represents a first quality, and the threshold rendering characteristic 160 represents a threshold quality. The first rendering characteristic 152 is a function of the first type of content represented by the first portion 50 of the previously rendered frame 30. For example, if the first portion 50 of the previously rendered frame 30 represents high frequency content, the first rendering characteristic 152 represents a relatively high resolution.

[0025] The content presentation system 200 renders the second portion 170 of the current frame 130, which corresponds to the second portion 70 of the previously rendered frame 30 (shown in FIG. 1B ), using second rendering characteristics 172. The second rendering characteristics 172 are less than the threshold rendering characteristics 160. In some implementations, the second rendering characteristics 172 are a second resolution that is lower than the first resolution of the first portion 150 of the current frame 130. In some implementations, the second rendering characteristics 172 are a second quality that is lower than the first quality of the first portion 150. The second rendering characteristics 172 are a function of the second type of content that the second portion 70 of the previously rendered frame 30 represents. For example, if the second portion 70 of the previously rendered content 30 represents infrequent content, the second rendering characteristics 172 represent a relatively low resolution.

[0026] The first portion 150 of the current frame 130 includes visual elements that may require the first rendering characteristics 152 to avoid adverse effects on visibility. For example, the building 32, the courtyard 34, the first tree 36, and the second tree 38 may need to be rendered at a sufficiently high resolution to produce an acceptable user experience. In contrast, the second portion 170 of the current frame 130 may include visual elements that can be rendered at the second rendering characteristics 172 without adversely affecting visibility. For example, the grass 40 and the clouds 42 may be rendered at a relatively lower resolution without adversely affecting the user experience. Rendering portions of the current frame 130 associated with low-frequency content at a relatively lower resolution tends to conserve computing resources, while simultaneously rendering portions of the current frame 130 associated with high-frequency content at a relatively higher resolution tends to provide an acceptable user experience. More generally, in various embodiments, different portions of the current frame 130 are rendered at respective resolutions based on their corresponding types of content. For example, infrequent content may be rendered at a low resolution, medium frequency content may be rendered at a medium resolution, and high frequency content may be rendered at a high resolution.

[0027] In some implementations, the content presentation system 200 generates the current frame 130 using techniques related to variable rate shading (VRS). For example, the content presentation system 200 may obtain (e.g., generate or receive) a map having different zones corresponding to respective portions of the current frame 130. Each zone of the map may indicate rendering characteristics for rendering the corresponding portion of the current frame 130. For example, a first zone of the map may indicate that a first portion 150 of the current frame 130 is to be rendered using first rendering characteristics 152, a second zone of the map may indicate that a second portion 170 of the current frame 130 is to be rendered using second rendering characteristics 172, a third zone of the map may indicate that a third portion of the current frame 130 is to be rendered using third rendering characteristics, and so on. The nth zone of the map may indicate that the nth portion of the current frame 130 is to be rendered using the nth rendering characteristics.

[0028] 2 is a block diagram of a content presentation system 200 according to some implementations. In some implementations, the content presentation system 200 resides in the electronic device 20 shown in FIGS. 1A-1C. In various implementations, the content presentation system 200 includes a data retriever 210, a content identifier 230, and a renderer 240.

[0029] In some implementations, the data obtainer 210 obtains the previously rendered frame 220 before the content presentation system 200 renders the current frame 250. For example, the data obtainer 210 obtains the previously rendered frame 30 shown in FIGS. 1A and 1B. In some implementations, the renderer 240 renders the previously rendered frame 220 at a previous time, and the data obtainer 210 receives the previously rendered frame 220 from the renderer 240. In some implementations, the previously rendered frame 220 corresponds to a different time period than the current frame 250. For example, the previously rendered frame 220 corresponds to a previous time period, and the current frame 250 corresponds to the current time period. Alternatively, in some implementations, the previously rendered frame 220 corresponds to the same time period as the current frame 250. For example, the previously rendered frame 220 may be a low-resolution rendering (e.g., a low-resolution version) of the current frame 250.

[0030] In various implementations, the content identifier 230 identifies the type of content represented by each portion of the previously rendered frame 220. In the example of FIG. 2, the content identifier 230 identifies that the first portion 222 of the previously rendered frame 220 represents a first type of content 232 and that the second portion 224 of the previously rendered frame 220 represents a second type of content 234. In some implementations, the first type of content 232 is associated with a first frequency set that is higher than a threshold frequency (e.g., first frequency 52 that is higher than threshold frequency 60 shown in FIG. 1B). Thus, in some implementations, the first type of content 232 is referred to as high frequency content. In some implementations, the second type of content 234 is associated with a second frequency set that is lower than a threshold frequency (e.g., second frequency 72 that is lower than threshold frequency 60 shown in FIG. 1B). Thus, in some implementations, the second type of content 234 is referred to as low frequency content.

[0031] In some implementations, the content identifier 230 performs a frequency analysis on the previously rendered frame 220 to determine a frequency associated with each portion of the previously rendered frame 220. In some implementations, the content identifier 230 generates a discrete cosine transform (DCT) of the previously rendered frame 220. The DCT indicates the frequency of each portion of the previously rendered frame 220. For example, the DCT indicates that a first portion 222 of the previously rendered frame 220 is associated with a relatively high frequency (e.g., a first frequency 52 that is higher than the threshold frequency 60 shown in FIG. 1B ) and a second portion 224 of the previously rendered frame 220 is associated with a relatively low frequency (e.g., a second frequency 72 that is lower than the threshold frequency 60 shown in FIG. 1B ).

[0032] In some implementations, the first type of content 232 refers to content having a first compression efficiency lower than a threshold compression efficiency, and the second type of content 234 refers to content having a second compression efficiency higher than the threshold compression efficiency. The first type of content 232 may be less compressible than the second type of content 232, for example, because the first type of content 232 may be associated with more texture values ​​than the second type of content 234. In some implementations, the content identifier 230 performs a compression analysis on the previously rendered frame 220 to determine the compression efficiency of each portion of the previously rendered frame 220. For example, the content identifier 230 compresses the previously rendered frame 220 and determines that the first portion 222 represents the first type of content 232 (e.g., content having a large number of texture values) due to a relatively low compression efficiency, and the second portion 224 represents the second type of content 234 (e.g., content having fewer texture values) due to a relatively high compression efficiency.

[0033] In some implementations, the content identifier 230 performs a reprojection operation on the previously rendered frame 220 before identifying the type of content associated with each portion of the previously rendered frame 220. In some implementations, the content identifier 230 obtains head pose data 236 indicative of the movement of the head of the user 12 since the previous head pose data was obtained (shown in FIGS. 1A-1C ). Reprojecting the previously rendered frame 220 increases the likelihood that the previously rendered frame 220 is similar to the current frame 250. Increasing the similarity between the previously rendered frame 220 and the current frame 250 tends to improve estimation of which portions of the current frame 250 are likely to correspond to high-frequency content.

[0034] In some implementations, the content identifier 230 indicates the position and / or size of the first portion 222 of the previously rendered frame 220 to the renderer 240. The renderer 240 renders the first portion 252 of the current frame 250 that corresponds to the first portion 222 of the previously rendered frame 220 using the first rendering characteristics 152 (e.g., the first rendering characteristics 262 shown in FIG. 1C ). The first rendering characteristics 262 are a function of the first type of content 232. In some implementations, the first rendering characteristics 262 are a first quality metric based on the first type of content 232. In some implementations, the first rendering characteristics 262 are a first resolution, and the first type of content 232 refers to a first frequency of the first portion 222 of the previously rendered frame 220. For example, the first rendering characteristic 262 is a relatively high resolution when the first type of content 232 is high frequency content (eg, content having a frequency higher than the threshold frequency 60 shown in FIG. 1B).

[0035] In some implementations, the content identifier 230 indicates the position and / or size of the second portion 224 of the previously rendered frame 220 to the renderer 240. The renderer 240 renders the second portion 254 of the current frame 250 corresponding to the second portion 224 of the previously rendered frame 220 using second rendering characteristics 172 (e.g., second rendering characteristics 264 shown in FIG. 1C ). The second rendering characteristics 264 are different from (e.g., smaller than or larger than) the first rendering characteristics 262. The second rendering characteristics 264 are a function of the second type of content 234. In some implementations, the second rendering characteristics 264 are a second quality metric based on the second type of content 234. In some implementations, the second rendering characteristics 264 are a second resolution, and the second type of content 234 refers to a second frequency of the second portion 224 of the previously rendered frame 220. For example, the second rendering characteristic 264 is a relatively low resolution when the second type of content 234 is low frequency content (eg, content having a frequency below the threshold frequency 60 shown in FIG. 1B).

[0036] In some implementations, the second portion 254 of the current frame 250 corresponds to a portion of the current frame 250 that is not encompassed by the first portion 252 of the current frame 250. For example, the renderer 240 renders the first portion 252 of the current frame 250 using the first rendering characteristics 262, and the renderer 240 renders the remaining portion of the current frame 250 using the second rendering characteristics 264. In some implementations, the second portion 254 of the current frame 250 refers to a portion of the current frame 250 that is expected to represent infrequent content.

[0037] In various implementations, the renderer 240 utilizes sparse rendering to render the current frame 250. In some implementations, the renderer 240 includes multiple renderers for rendering different portions of the current frame 250 associated with respective rendering characteristics. For example, the renderer 240 includes a first renderer that renders the first portion 252 of the current frame 250 using first rendering characteristics 262 and a second renderer that renders the second portion 254 of the current frame 250 using second rendering characteristics 264.

[0038] In some implementations, the renderer 240 utilizes different shaders to render different portions of the current frame 250. In some implementations, the renderer 240 utilizes shaders of varying complexity to render different portions of the current frame 250. In some implementations, the complexity of the shader that the renderer 240 utilizes to render a particular portion of the current frame 250 is based on the rendering characteristics used when that particular portion of the current frame 250 is rendered. For example, the renderer 240 utilizes a first shader having a first level of complexity (e.g., relatively high complexity) to render the first portion 252 of the current frame 250 and a second shader having a second level of complexity (e.g., relatively low complexity) to render the second portion 254 of the current frame 250. In some implementations, the complexity level of the shader that the renderer 240 utilizes to render a portion of the current frame 250 is proportional to the value of the rendering characteristics with which that portion of the current frame 250 is to be rendered.

[0039] In some implementations, the renderer 240 utilizes a tree data structure to represent different portions of the current frame 250. In some implementations, the renderer 240 utilizes a quadtree representation to represent different portions of the current frame 250. For example, each quadrant in the quadtree representation may correspond to a particular portion of the current frame 250. In some implementations, the quadrant to which a particular portion of the current frame 250 is mapped is based on the corresponding rendering characteristics of that particular portion of the current frame 250.

[0040] In some implementations, rendering the second portion 254 of the current frame 250 using the second rendering characteristics 264 requires fewer computing resources than rendering the first portion 252 of the current frame 250 using the first rendering characteristics 262. Because the first portion 252 of the current frame 250 corresponds to the first portion 222 of the previously rendered frame 220 and the second portion 254 of the current frame 250 corresponds to the second portion 224 of the previously rendered frame 220, identifying the first portion 222 and second portion 224 of the previously rendered frame 220 allows the renderer 240 to conserve resources by rendering the second portion 254 using the second rendering characteristics 264 and to produce an acceptable user experience by rendering the first portion 252 using the first rendering characteristics 262.

[0041] 3 is a flowchart representation of a method 300 for foveating frames based on content. In some implementations, method 300 is performed by a device including a display, non-transitory memory, and one or more processors coupled to the display and the non-transitory memory. In various implementations, method 300 is performed by electronic device 20 shown in FIGS. 1A-1C and / or content presentation system 200 shown in FIGS. 1A-2. In some implementations, method 300 is performed by processing logic, which includes hardware, firmware, software, or a combination thereof. In some implementations, method 300 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory).

[0042] As represented by block 310, in various implementations, method 300 includes obtaining a previously rendered frame before rendering a current frame. For example, as shown in FIGS. 1A-1C, electronic device 20 obtains previously rendered frame 30 before rendering current frame 130. As another example, as shown in FIG. 2, content presentation system 200 obtains previously rendered frame 220 before rendering current frame 250. In some implementations, method 300 includes receiving the previously rendered frame from a renderer that rendered the previously rendered frame at a previous time. In some implementations, method 300 includes retrieving the previously rendered frame from a memory location.

[0043] As represented by block 310a, in some implementations, the current frame corresponds to a first time and the previously rendered frame corresponds to a second time that occurred before the first time. For example, the current frame corresponds to the current time and the previously rendered frame corresponds to a previous time. In some implementations, the time difference between the first time corresponding to the current frame and the second time corresponding to the previously rendered frame is less than a threshold. Limiting the time difference between the first time and the second time tends to reduce the difference in content between the previously rendered frame and the current frame. Limiting the difference in content between the previously rendered frame and the current frame allows the device to use the previously rendered frame to predict which portions of the current frame will be rendered at a relatively high resolution and which portions of the current frame may be rendered at a relatively low resolution.

[0044] As represented by block 310b, in some implementations, the previously rendered frame is a low-resolution rendering of the current frame. For example, the previously rendered frame 30 shown in FIGS. 1A and 1B may be a low-resolution version of the current frame 130 shown in FIG. 1C. As another example, the previously rendered frame 220 shown in FIG. 2 may be a low-resolution version of the current frame 250. In some implementations, the device renders a low-resolution version of the current frame. The device analyzes the low-resolution version of the current frame to identify portions that represent frequent content. After identifying the portions that represent frequent content, the device can render the portions that represent frequent content at high resolution and render the remainder of the current frame at low resolution. Rendering a low-resolution version of the current frame is less resource-intensive than rendering the entire current frame at high resolution.

[0045] As represented by block 320, in various implementations, the method 300 includes determining that a first portion of a previously rendered frame is associated with a particular type of content. For example, as shown in FIG. 2, the content identifier 230 determines that the first portion 222 of the previously rendered frame 220 represents a first type of content. In some implementations, the method 300 includes identifying various portions (e.g., all portions) of the previously rendered frame that represent the particular type of content. In some implementations, the method 300 includes performing image analysis on the previously rendered frame to determine the type of content represented by each portion of the previously rendered frame. In some implementations, the method 300 includes performing a frequency analysis on the previously rendered frame to determine a frequency associated with each portion of the previously rendered frame.

[0046] As represented by block 320a, in some implementations, before determining that a first portion of the previously rendered frame is associated with a particular type of content, the method 300 includes reprojecting the previously rendered frame based on head pose data indicative of a head pose of a user of the device. For example, as shown in FIG. 2, the content identifier 230 reprojects the previously rendered frame 220 based on the head pose data 236. Reprojecting the previously rendered frame 220 may include warping the previously rendered frame 220 based on the head movement indicated by the head pose data 236. Reprojecting the previously rendered frame tends to reduce the differences between the previously rendered frame and the current frame to be rendered, thereby making the content represented by the previously rendered frame an acceptable approximation of the content that the current frame is likely to represent.

[0047] As represented by block 320b, in some implementations, method 300 includes applying a set of one or more filters to the previously rendered frame before determining that a first portion of the previously rendered frame is associated with a particular type of content. For example, with reference to FIG. 2, content identifier 230 may apply a set of one or more filters to previously rendered frame 220 before determining that first portion 222 of previously rendered frame 220 represents a first type of content 232 and that second portion 224 of previously rendered frame 220 represents a second type of content 234. In some implementations, applying a filter(s) to the previously rendered frame tends to reduce differences between the previously rendered frame and the current frame to be rendered, thereby making the content of the previously rendered frame an acceptable prediction of the content of the current frame.

[0048] As represented by block 320c, in some implementations, determining that a first portion of a previously rendered frame is associated with a particular type of content includes determining that the first portion of a previously rendered frame is associated with a first frequency that is higher than a threshold frequency. For example, as shown in FIG. 1B, content presentation system 200 determines that a first portion 50 of a previously rendered frame 30 represents content associated with a first frequency 52 that is higher than a threshold frequency 60. In some implementations, method 300 includes determining that a second portion of a previously rendered frame is associated with a second frequency that is lower than the threshold frequency. For example, as shown in FIG. 1B, content presentation system 200 determines that a second portion 70 of a previously rendered frame 30 represents content associated with a second frequency 72 that is lower than the threshold frequency 60. In some implementations, the device determines that the first portion of a previously rendered frame represents high-frequency content and that the second portion of a previously rendered frame represents low-frequency content.

[0049] In some implementations, the method 300 includes performing a discrete cosine transform (DCT) on the previously rendered frame to determine that a first portion of the previously rendered frame is associated with a first frequency and a second portion of the previously rendered frame is associated with a second frequency. In some implementations, the DCT of the previously rendered frame indicates that the first portion of the previously rendered frame is associated with a first range of frequencies higher than a threshold frequency and the second portion of the previously rendered frame is associated with a second range of frequencies lower than the threshold frequency. In some implementations, the DCT of the previously rendered frame indicates that the first portion of the previously rendered frame represents high-frequency content and the second portion of the previously rendered frame represents low-frequency content.

[0050] As represented by block 320d, in some implementations, determining that a first portion of a previously rendered frame is associated with a particular type of content includes determining that the compression efficiency of the first portion is lower than a compression efficiency threshold. In some implementations, method 300 includes compressing a previously rendered frame and determining a compression efficiency for each portion of the previously rendered frame. Portions of a previously rendered frame that can be compressed with a relatively high compression efficiency are likely to be associated with a relatively small number of texture values, and therefore, the corresponding portions of the current frame can be displayed at a relatively low resolution, thereby conserving rendering resources. In contrast, portions of a previously rendered frame that are compressed with a relatively low compression efficiency are likely to be associated with a relatively large number of texture values, and therefore, the corresponding portions of the current frame need to be displayed at a relatively high resolution to provide an acceptable level of user experience.

[0051] In some implementations, determining that a first portion of a previously rendered frame is associated with a particular type of content includes determining that the number of triangles rendered associated with the first portion is greater than a threshold number of triangles. In some implementations, method 300 includes determining the number of polygons rendered associated with (e.g., within) each portion of a previously rendered frame. For example, the device determines a number of triangles rendered within each portion of a previously rendered frame. Portions of a previously rendered frame that include a relatively large number of triangles are likely to correspond to portions of a current frame that will be rendered at a relatively high resolution to produce an acceptable level of user experience. In contrast, portions of a previously rendered frame that include a relatively small number of triangles are likely to correspond to portions of a current frame that can be rendered at a relatively low resolution to conserve rendering resources without sacrificing user experience.

[0052] As represented by block 330, in some implementations, the method 300 includes rendering a first portion of a current frame corresponding to a first portion of a previously rendered frame using first rendering characteristics and rendering a second portion of the current frame using second rendering characteristics that are different from the first rendering characteristics. For example, as shown in FIG. 1C , the content presentation system 200 renders a first portion 150 of the current frame 130 using first rendering characteristics 152 that are greater than a threshold rendering characteristic 160, and the content presentation system 200 renders a second portion 170 of the current frame 130 using second rendering characteristics 172 that are less than the threshold rendering characteristic 160. Rendering the first portion of the current frame using the first rendering characteristics enables the device to display the first portion of the current frame using target visibility (e.g., with an acceptable level of user experience). Because rendering the content using the second rendering characteristic requires fewer rendering resources than rendering the content using the first rendering characteristic, not rendering the second portion of the current frame using the first rendering characteristic allows the device to conserve rendering resources.

[0053] As represented by block 330a, in some implementations, the first rendering characteristic corresponds to a first resolution, and the second rendering characteristic corresponds to a second resolution lower than the first resolution. For example, as described in connection with FIG. 1C , the first rendering characteristic 152 at which the first portion 150 of the current frame 130 is rendered may refer to a first resolution higher than the threshold resolution represented by the threshold rendering characteristic 160, and the second rendering characteristic 172 at which the second portion 170 of the current frame 130 is rendered may refer to a second resolution lower than the threshold resolution represented by the threshold rendering characteristic 160. Rendering content at the first resolution may be more resource-intensive than rendering content at the second resolution. Therefore, not rendering the second portion of the current frame at the first resolution tends to conserve rendering resources.

[0054] In some implementations, the first rendering characteristic corresponds to a first quality, and the second rendering characteristic corresponds to a second quality that is lower than the first quality. For example, with reference to FIG. 2 , rendering the first portion 252 of the current frame 250 using the first rendering characteristic 262 may refer to rendering the first portion 252 of the current frame 250 at a first quality, and rendering the second portion 254 of the current frame 250 using the second rendering characteristic 264 may refer to rendering the second portion 254 of the current frame 250 at a second quality that is lower than the first quality. Rendering content at the first quality may require more rendering resources than rendering content at the second quality. Thus, rendering the second portion of the current frame at the second quality tends to conserve rendering resources.

[0055] As represented by block 330b, in some implementations, the second portion of the current frame corresponds to a second portion of a previously rendered frame that is not associated with a particular type of content. For example, as shown in FIGS. 1B and 1C, the second portion 170 of the current frame 130 corresponds to a second portion 70 of the previously rendered frame 30 that is associated with a second frequency 72 that is lower than the threshold frequency 60. In some implementations, the method 300 includes estimating, based on the first portion of the previously rendered frame, that the first portion of the current frame represents high-frequency content that needs to be rendered at a first resolution. In some implementations, the method 300 includes estimating, based on the second portion of the previously rendered frame, that the second portion of the current frame represents low-frequency content that may be rendered at a second resolution that is lower than the first resolution.

[0056] As represented by block 330c, in various implementations, the method 300 includes utilizing sparse rendering to render different portions of the current frame with respective rendering characteristics. In some implementations, a first portion is rendered by a first renderer that renders content at a first resolution, and a second portion is rendered by a second renderer that renders content at a second resolution that is lower than the first resolution. For example, referring to FIG. 2, the renderer 240 may include a high-resolution renderer that renders a first portion 252 of the current frame 250 at a relatively high resolution and a low-resolution renderer that renders a second portion 254 of the current frame 250 at a relatively low resolution.

[0057] In some implementations, rendering the first portion includes utilizing a first shader associated with a first complexity level, and rendering the second portion includes utilizing a second shader associated with a second complexity level lower than the first complexity level. In some implementations, rendering the first portion at a relatively high resolution includes utilizing a shader of relatively high complexity, and rendering the second portion at a relatively low resolution includes utilizing a shader of relatively low complexity. In some implementations, the complexity of the shader is a function of rendering characteristics. For example, rendering content at a relatively high resolution utilizes a shader of higher complexity, and rendering content at a relatively low resolution utilizes a shader of lower complexity.

[0058] In some implementations, method 300 includes utilizing a quadtree data structure to represent a first portion of the current frame and a second portion of the current frame. In some implementations, each quadrant of the quadtree data structure represents a portion of the current frame to be rendered with particular rendering characteristics. In some implementations, the quadtree data structure represents a hierarchy of portions of the current frame, with portions of the current frame at higher tiers in the hierarchy being rendered with greater rendering characteristics than portions of the current frame at lower tiers in the hierarchy.

[0059] FIG. 4 is a block diagram of a device 400 according to some embodiments. In some implementations, the device 400 implements the electronic device 20 shown in FIGS. 1A-1C and / or the content presentation system 200 shown in FIGS. 1A-2. While certain individual features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the implementations disclosed herein. To that end, by way of non-limiting example, in some implementations, the device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, memory 404, one or more input / output (I / O) devices 408, and one or more communication buses 405 for interconnecting these and various other components.

[0060] In some implementations, network interface 402 is provided for, among other uses, establishing and maintaining a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compliant devices. In some implementations, one or more communication buses 405 include circuitry for interconnecting and controlling communications between system components. Memory 404 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access semiconductor memory devices, and may also include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices located remotely from one or more CPUs 401. Memory 404 includes a non-transitory computer-readable storage medium.

[0061] In some implementations, one or more I / O devices 408 include a display (e.g., display 22 shown in FIGS. 1A-1C). In some implementations, the display includes an extended reality (XR) display. In some implementations, the display includes an opaque display. Alternatively, in some implementations, the display includes an optical see-through display. In some implementations, one or more I / O devices 408 include an image sensor.

[0062] In some implementations, the memory 404, or the non-transitory computer-readable storage medium of the memory 404, stores programs, modules and data structures, or a subset thereof, such as an optional operating system 406, a data retriever 210, a content identifier 230, and a renderer 240.

[0063] In various implementations, the data acquirer 210 includes instructions 210a and heuristics and metadata 210b for acquiring a previously rendered frame (e.g., previously rendered frame 30 shown in FIGS. 1A and 1B and / or previously rendered frame 220 shown in FIG. 2) before rendering the current frame.

[0064] In some implementations, the content identifier 230 includes instructions 230a for determining that a first portion of a previously rendered frame is associated with a particular type of content (e.g., for determining that a first portion 50 of a previously rendered frame 30 represents content associated with a first frequency 52 higher than the threshold frequency 60 shown in FIG. 1B ) and heuristics and metadata 230b.

[0065] In some implementations, the renderer 240 includes instructions 240a for rendering a first portion of a current frame that corresponds to a first portion of a previously rendered frame using first rendering characteristics and rendering a second portion of the current frame using second rendering characteristics that are different from the first rendering characteristics (e.g., for rendering a first portion 150 of the current frame 130 using first rendering characteristics 152 that are greater than the threshold rendering characteristics 160 and rendering a second portion 170 of the current frame 130 using second rendering characteristics 172 that are less than the threshold rendering characteristics 160), and heuristics and metadata 240b.

[0066] It will be appreciated that Figure 4 is not an architectural overview of the implementations described herein, but rather is intended to illustrate the functionality of various features that may be present in particular implementations. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional blocks shown separately in Figure 4 can be implemented as a single block, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of blocks, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some implementations, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0067] While various aspects of implementations within the scope of the appended claims have been described above, it should be apparent that various features of the above-described implementations may be embodied in a wide variety of forms, and that any specific structure and / or function described above is merely illustrative. Based on this disclosure, one skilled in the art will understand that any aspect described herein may be practiced independently of any other aspect, or that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and / or a method may be practiced using any number of the aspects described herein. In addition, such an apparatus may be implemented and / or such a method may be practiced using structure and / or functionality in addition to one or more of the aspects described herein.

Claims

1. 1. A device including a display, non-transitory memory, and one or more processors coupled to the display and the non-transitory memory, Retrieving the previously rendered frame before rendering the current frame; determining that a first portion of the previously rendered frame is associated with a particular type of content; rendering a first portion of the current frame corresponding to the first portion of the previously rendered frame with first rendering characteristics, and rendering a second portion of the current frame with second rendering characteristics different from the first rendering characteristics.

2. The method of claim 1 , wherein the current frame corresponds to a first time instant and the previously rendered frame corresponds to a second time instant that occurred before the first time instant.

3. The method of claim 1 , wherein the previously rendered frame is a low-resolution rendering of the current frame.

4. 4. The method of claim 1, further comprising: before determining that the first portion of the previously rendered frame is associated with the particular type of content, reprojecting the previously rendered frame based on head pose data indicative of a head pose of a user of the device.

5. 5. The method of claim 1, further comprising applying a set of one or more filters to the previously rendered frame before determining that the first portion of the previously rendered frame is associated with the particular type of content.

6. 6. The method of claim 1, wherein determining that the first portion of the previously rendered frame is associated with the particular type of content comprises determining that the first portion of the previously rendered frame is associated with a first frequency that is higher than a threshold frequency.

7. The method of claim 1 , further comprising determining that a second portion of the previously rendered frame is associated with a second frequency that is lower than the threshold frequency.

8. 8. The method of claim 7, further comprising generating a discrete cosine transform (DCT) for the previously rendered frame to determine that the first portion of the previously rendered frame is associated with the first frequency and the second portion of the previously rendered frame is associated with the second frequency.

9. 9. The method of claim 1, wherein determining that the first portion of the previously rendered frame is associated with the particular type of content comprises determining that a compression efficiency of the first portion is below a compression efficiency threshold.

10. 10. The method of claim 1, wherein determining that the first portion of the previously rendered frame is associated with the particular type of content comprises determining that a number of triangles rendered in association with the first portion is greater than a threshold number of triangles.

11. The method of claim 1 , wherein the first rendering characteristic corresponds to a first resolution and the second rendering characteristic corresponds to a second resolution that is lower than the first resolution.

12. The method of claim 1 , wherein the first rendering characteristic corresponds to a first quality and the second rendering characteristic corresponds to a second quality that is lower than the first quality.

13. 13. The method of claim 1, wherein the second portion of the current frame corresponds to a second portion of the previously rendered frame that is not associated with the particular type of content.

14. 14. The method of claim 1, wherein the first portion of the current frame is rendered by a first renderer that renders content at a first resolution, and the second portion of the current frame is rendered by a second renderer that renders content at a second resolution that is lower than the first resolution.

15. 15. The method of claim 1, wherein rendering the first portion of the current frame comprises utilizing a first shader associated with a first complexity level, and wherein rendering the second portion of the current frame comprises utilizing a second shader associated with a second complexity level that is lower than the first complexity level.

16. The method of claim 1 , further comprising utilizing a quadtree data structure to represent the first portion of the current frame and the second portion of the current frame.

17. The display and one or more processors; a non-transient memory; one or more programs stored in the non-transitory memory, the one or more programs, when executed by the one or more processors, causing the device to perform the method of any one of claims 1 to 16; 1. A device comprising:

18. 17. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device having a display, cause the device to perform the method of any one of claims 1 to 16.

19. The display and one or more processors; a non-transient memory; means for causing said device to carry out a method according to any one of claims 1 to 16; A device comprising:

20. The display and one or more processors; a non-transient memory; one or more programs stored in the non-transitory memory that, when executed by the one or more processors, cause the device to: Before rendering the current frame, retrieve the previously rendered frame, determining that a first portion of the previously rendered frame is associated with a particular type of content; and one or more programs that cause a first portion of the current frame corresponding to the first portion of the previously rendered frame to be rendered with first rendering characteristics, and that cause a second portion of the current frame to be rendered with second rendering characteristics that are different from the first rendering characteristics.

21. a non-transitory memory storing one or more programs, the one or more programs, when executed by one or more processors of a device having a display, causing the device to: Before rendering the current frame, retrieve the previously rendered frame, determining that a first portion of the previously rendered frame is associated with a particular type of content; a non-transitory memory that causes a first portion of the current frame corresponding to the first portion of the previously rendered frame to be rendered with first rendering characteristics and a second portion of the current frame to be rendered with second rendering characteristics different from the first rendering characteristics;

Images