Reducing energy consumption in augmented reality devices

Abstract

Reducing energy consumption in an augmented reality device is disclosed. A method implemented by a computing device includes displaying an augmented reality (XR) environment on a display of the computing device, and determining a context of the XR environment. Determining the context includes determining one or more features associated with virtual content and a real-world content scene that are included within the displayed XR environment. The method further includes determining, based on the one or more features, that the virtual content is highlighted relative to the real-world content scene, and in response to determining that the virtual content is highlighted relative to the real-world content scene, abandoning image processing of the real-world content scene.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. non-provisional patent application No. 18 / 458,682, filed on August 30, 2023, which is incorporated by reference into this application. Technical Field

[0003] The present disclosure relates generally to extended reality devices, and more particularly to reducing energy consumption in extended reality devices. Background Art

[0004] An extended reality (XR) system may typically include a real-world environment that includes XR content that overlays one or more real-world environment features. In a typical XR system, image data may be presented on, for example, a robust head-mounted display (HMD), which may be coupled to a basic graphics generation device responsible for generating image data via a physical wired connection or a wireless connection. However, in some instances where the HMD includes, for example, lightweight XR glasses and / or other wearable electronic devices rather than a more robust head-mounted viewer (headset) device, in contrast, the XR glasses or other lightweight wearable electronic devices may include reduced processing power, low or high resolution cameras, and / or relatively simple tracking optics. In addition, due to a smaller manufacturing area, the XR glasses or other lightweight wearable electronic devices may also include reduced power management (e.g., battery, battery size) electronics and thermal management (e.g., cooling fans, radiators) electronics. This typically prevents such devices from maximizing performance while reducing power consumption and thermal impact. Therefore, it may be useful to provide techniques for improving XR systems. Summary of the invention

[0005] The present embodiment includes techniques for dynamically abandoning image processing of a real-world content scene obscured by virtual content and / or deactivating one or more image sensors of a computing device corresponding to the capture of a real-world content scene obscured by virtual content. In some embodiments, a computing device may display an extended reality (XR) environment on a display of the computing device. In some embodiments, the computing device may determine a context of the XR environment. For example, in one embodiment, determining the context may include determining one or more features associated with virtual content and real-world content scenes that are included within the displayed XR environment. In some embodiments, determining one or more features may include determining the amount of one or more of shadows, occlusions, collisions, diffuse reflections, or specular reflections associated with the virtual content and the real-world content scene.

[0006] In some embodiments, the computing device may include determining that the virtual content is highlighted relative to the real-world content scene based on one or more features. In one embodiment, determining that the virtual content is highlighted relative to the real-world content scene based on one or more features may include determining that the virtual content obscures a portion of the real-world content scene. In another embodiment, determining that the virtual content is highlighted relative to the real-world content scene based on one or more features may include determining that the virtual content completely obscures the real-world content scene. In some embodiments, in response to determining that the virtual content is highlighted relative to the real-world content scene, the computing device may then deactivate one or more image sensors of the computing device.

[0007] For example, in some embodiments, one or more image sensors may be configured to capture a real-world content scene. In some embodiments, the one or more image sensors include one or more color image sensors, one or more monochrome image sensors, or one or more depth image sensors. In some embodiments, deactivating one or more image sensors may include: deactivating at least one of the one or more image sensors. For example, at least one of the one or more image sensors may be configured to capture an obscured portion of a real-world content scene. In some embodiments, deactivating one or more image sensors may also include: deactivating the one or more image sensors when the virtual content completely obscures the real-world content scene, and restarting at least one of the one or more image sensors when the virtual content does not completely obstruct the real-world content scene. In some embodiments, in response to determining that the virtual content is highlighted relative to the real-world content scene, the computing device may redisplay the XR environment without the real-world content scene on the display. In this way, the present embodiment can reduce the energy consumption (e.g., power consumption, thermal impact) of the computing device by dynamically abandoning image processing of the real-world content scene obscured by the virtual content and / or deactivating one or more image sensors of the computing device corresponding to the capture of the real-world content scene obscured by the virtual content.

[0008] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. Certain embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps in the embodiments disclosed above. Embodiments according to the present invention are particularly disclosed in the attached claims relating to methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or backreferences in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentionally backreferenced any previous claim (especially multiple dependencies) may also be claimed, so that any combination of multiple claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any embodiment and any feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or any feature described or depicted herein or in any combination with any feature in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example embodiment of an extended reality (XR) system is shown.

[0010] Figure 2 An example embodiment of an XR environment is shown.

[0011] FIG. 3A to FIG. 3F One or more operational examples of deactivating one or more image sensors of a computing device corresponding to capturing of a real-world content scene occluded by virtual content, or abandoning image processing of a real-world content scene occluded by virtual content are shown.

[0012] Figure 4 A flow chart of a method for deactivating one or more image sensors of a computing device corresponding to capturing a real-world content scene occluded by virtual content, or forgoing image processing of a real-world content scene occluded by virtual content, is shown.

[0013] Figure 5 An example network environment associated with a virtual reality system is shown.

[0014] Figure 6 An example computer system is shown. DETAILED DESCRIPTION

[0015] An extended reality (XR) system may typically include a real-world environment that includes XR content that overlays one or more real-world environment features. In a typical XR system, image data may be presented on, for example, a robust head-mounted display (HMD), which may be coupled to a basic graphics generation device responsible for generating image data via a physical wired connection or a wireless connection. However, in some instances where the HMD includes, for example, lightweight XR glasses and / or other wearable electronic devices rather than a more robust head-mounted viewer device, the XR glasses or other lightweight wearable electronic devices may include reduced processing power, low or high resolution cameras, and / or relatively simple tracking optics in comparison. In addition, due to a smaller manufacturing area, the XR glasses or other lightweight wearable electronic devices may also include reduced power management (e.g., battery, battery size) electronics and thermal management (e.g., cooling fans, radiators) electronics. This typically prevents such devices from maximizing performance while reducing power consumption and thermal impact. Therefore, it may be useful to provide techniques for improving XR systems.

[0016] Therefore, the present embodiment includes techniques for dynamically abandoning image processing of a real-world content scene obscured by virtual content and / or deactivating one or more image sensors in a computing device corresponding to the capture of a real-world content scene obscured by virtual content. In some embodiments, a computing device may display an extended reality (XR) environment on a display of the computing device. In some embodiments, the computing device may determine a context of the XR environment. For example, in one embodiment, determining the context may include determining one or more features associated with virtual content and real-world content scenes that are included within the displayed XR environment. In some embodiments, determining the one or more features may include determining the amount of one or more of shadows, occlusions, collisions, diffuse reflections, or specular reflections associated with the virtual content and the real-world content scene.

[0017] In some embodiments, the computing device may include determining that the virtual content is highlighted relative to the real-world content scene based on one or more features. In one embodiment, determining that the virtual content is highlighted relative to the real-world content scene based on one or more features may include determining that the virtual content obscures a portion of the real-world content scene. In another embodiment, determining that the virtual content is highlighted relative to the real-world content scene based on one or more features may include determining that the virtual content completely obscures the real-world content scene. In some embodiments, in response to determining that the virtual content is highlighted relative to the real-world content scene, the computing device may then deactivate one or more image sensors of the computing device.

[0018] For example, in some embodiments, one or more image sensors may be configured to capture a real-world content scene. In some embodiments, the one or more image sensors include one or more color image sensors, one or more monochrome image sensors, or one or more depth image sensors. In some embodiments, deactivating one or more image sensors may include deactivating at least one of the one or more image sensors. For example, at least one of the one or more image sensors may be configured to capture an obscured portion of a real-world content scene. In some embodiments, deactivating one or more image sensors may also include deactivating the one or more image sensors when the virtual content completely obscures the real-world content scene, and restarting at least one of the one or more image sensors when the virtual content does not completely obstruct the real-world content scene. In some embodiments, in response to determining that the virtual content is highlighted relative to the real-world content scene, the computing device may redisplay the XR environment without the real-world content scene on the display. In this way, the present embodiment can reduce the energy consumption (e.g., power consumption, thermal impact) of the computing device by dynamically abandoning image processing of the real-world content scene obscured by the virtual content and / or by deactivating one or more image sensors of the computing device corresponding to the capture of the real-world content scene obscured by the virtual content.

[0019] As used herein, "extended reality" may refer to an electronic-based form of reality that has been manipulated in some way before being presented to a user, including, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, simulated reality, immersive reality, hologram, or any combination thereof. For example, "extended reality" content may include fully computer-generated content, or partially computer-generated content combined with captured content (e.g., images of the real world). In some embodiments, "extended reality" content may also include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional (3D) effect). In addition, as used herein, it should be understood that "extended reality" may be associated with applications, products, accessories, services, or combinations thereof, which may, for example, be used to create content in extended reality and / or be used in extended reality (e.g., performing activities in extended reality). Thus, “extended reality” content may be implemented on a variety of platforms, including a head-mounted device (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing extended reality content to one or more viewers.

[0020] Figure 1 A cross-section of an example XR display device 100 is shown. The XR display device 100 includes an example wearable display 110, which may include at least one waveguide 115. It should be understood that the XR display device 100 as shown is an example of one embodiment of a head-mounted display (HMD) that can be used to reduce energy consumption in accordance with the presently disclosed embodiments. In another embodiment, the XR display device 100 may include a see-through HMD that may not include a waveguide and may present an image directly on, for example, one or more transparent or semi-transparent mirrors that may be placed in front of the user's eyes. Figure 1Also shown is an eye zone 122, which is where the user's eye 120 can be positioned relative to the display 110 when the user wears the XR display device 100. For example, as long as the user's eye 120 is aligned with the eye zone 122, the user can see a full-color image, or a pupil replication toward the eye zone 122 through the waveguide 115. The waveguide 115 can generate many pupil replications and guide these pupil replications to the eye zone 122. The waveguide 115 can be configured to guide the image light 160 to the eye zone 122 located near the user's eye 120. For illustration, Figure 1 A cross section is shown associated with a single eye 120 of a user and a single waveguide 115. In some embodiments, waveguide 115 or another waveguide may provide image light to an ocular region located at the user's other eye.

[0021] In some embodiments, the waveguide 115 can be composed of one or more materials (e.g., plastic, glass, etc.) having one or more refractive indices that effectively minimize the weight of the display 110 and expand the field of view (FOV) of the display 110. In one or more embodiments, the display 110 can include one or more optical elements located between the waveguide 115 and the user's eye 120. The optical elements can be used, for example, to correct aberrations in the image light 160, amplify the image light 160, make some other optical adjustments to the image light 160, or perform a combination of the above operations. Examples of optical elements can include an aperture, a Fresnel lens, a refractive lens (e.g., a convex lens and / or a concave lens), a reflective surface, a filter, or any other suitable optical element that affects the image light. For example, the waveguide 115 can include a waveguide having one or more sets of Bragg gratings.

[0022] In some embodiments, the display 110 may include a scanline or one-dimensional ("one-dimensional, 1D") waveguide display. In such an embodiment, a row of light sources may generate light for illuminating the entire vertical space (or horizontal space when appropriate) of the display. Multiple smaller images may be combined to form a larger composite image perceived by a viewer. The scanning element may cause the source light processed by the waveguide component to be output to the user's eye 120 of the user in a specific pattern as follows to optimize the display drawing rate: the specific pattern corresponds to the generation pattern used by the emitter. For example, the light source may first be provided with color values ​​corresponding to a single row of pixels along the top of the displayed image.

[0023] In some embodiments, a waveguide-based process assisted by an oscillating mirror powered by a microelectromechanical system (MEMS) can be used to transmit light to the appropriate portion of the eye zone 122. After a short period of time, the light source can be provided with color values ​​corresponding to the next row of pixels (e.g., the next row of pixels below the first row). The light for this portion of the image can then be positioned at the appropriate location using the same method. Scanning displays can operate using lower power and can generate less heat than traditional displays composed of the same emitters. Scanning displays can also have a lighter weight, in part due to the amount of material used in the scanning elements and optical system. The display's

[0024] In other embodiments, the display 110 may include a 2D or two-dimensional waveguide display. In such a display, an oscillating mirror is not used because a light source including a vertical component and a horizontal component can be used (e.g., in an array). Where the 1D variant illuminates the display row by row, the 2D variant may be able to provide a significantly improved frame rate because the 2D variant does not rely on an oscillating mirror to provide the vertical component of the image. To further increase the frame rate, the light source of the 2D waveguide display may be bonded to a controller and / or memory that provides drive instructions for the display system. For example, the light source may be bonded to a memory that stores color instructions for the display and / or driver transistors. As a result of this configuration, the light source for such a display may be able to operate at a significantly faster frame rate.

[0025] In some embodiments, the XR display device 100 may include a light source such as a projector 112 that emits projection light 155 depicting one or more images. Many suitable display light source technologies are contemplated, including but not limited to: liquid crystal display (LCD), liquid crystal on silicon (LCOS), light-emitting diode (LED), organic LED (OLED), micro-LED (μLED), digital micromirror device (DMD), any other suitable display technology, or any combination of the above technologies. The projection light 155 may be received by the first coupler 150 of the waveguide 115. The waveguide 115 may combine the projection light 155 with a real-world scene 116 (e.g., scene light) received by the second coupler 152. The real-world scene 116 (e.g., scene light) may be, for example, light from a real-world environment and may pass through the transparent (or translucent) surface 154 to the second coupler 152. The transparent surface 154 may be, for example, a protective curved glass or a lens formed of glass, plastic, or other transparent material.

[0026] In some embodiments, the coupling components of the waveguide 115 can guide the projection light 155 along the total internal reflection path of the waveguide 115. In addition, the projection light 155 can first pass through a small air gap between the projector 112 and the waveguide 115, and then interact with a coupling element (e.g., the first coupler 150) incorporated into the waveguide. In some examples, the optical path can include a grating structure or other type of optical decoupling structure that decouples part of the light from the total internal reflection path to guide multiple instances of the image ("pupil replicas") from different locations of the waveguide 115 and directed to the eye zone 122 of the XR display device 100.

[0027] In some embodiments, scene light 116 may be visible to user eye 120. For example, Figure 1As further shown, the XR display device 100 may include one or more cameras 126A and 126B. In certain embodiments, the one or more cameras 126A and 126B may include one or more color cameras (e.g., RGB (red, green, blue) cameras), one or more monochrome cameras, or one or more color depth cameras 126B (e.g., RGB-D depth cameras), which may be suitable for detecting or capturing the real-world scene 116 (e.g., scene light) and / or certain features of the real-world scene 116 (e.g., scene light). For example, in some embodiments, in order to provide an XR experience to a user, the one or more cameras 126A and 126B may include a high-resolution RGB image sensor, which may potentially be continuously “on” (e.g., activated), for example, during the time that the user spends in extended reality. Because the one or more cameras 126A and 126B are continuously “on” (e.g., activated), the one or more cameras 126A and 126B may have an adverse impact on the overall power consumption and thermal rating of the XR display device 100. Reference will be made to FIG. 3A to FIG. 3F as well as Figure 4 It is further understood that the present embodiment may include dynamically deactivating one or more cameras 126A and 126B corresponding to capturing of a real-world content scene occluded by virtual content or forgoing image processing of a real-world content scene occluded by virtual content.

[0028] In some embodiments, one or more controllers 130 can control the operation of the projector 112 and the plurality of cameras 126A and 126B. The controller 130 can generate display instructions for the display system of the projector 112 or image capture instructions for the one or more cameras 126A and 126B. For example, the display instructions may include instructions to project or transmit one or more images, and the image capture instructions may include instructions to capture one or more images in a continuous sequence. In some embodiments, the display instructions and the image capture instructions may include frame image color or monochrome data. The display instructions and the image capture instructions may be received from, for example, one or more processing devices, which are included in Figure 1 The controller 130 may include a combination of hardware, software, and / or firmware, which is not explicitly shown herein to avoid obscuring other aspects of the present disclosure.

[0029] Figure 2 An example isometric view of an XR environment 200 is shown. In some embodiments, the XR environment 200 may be a component of the XR display device 100. The XR environment 200 may include at least one projector 112, a waveguide 115, and a controller 130. The content renderer 132 may generate a representation of content (the content is referred to herein as AR virtual content 157) to be projected by the projector 112 as projection light 155. The content renderer 132 may send the representation of the content to the controller 130, which in turn may generate display instructions based on the content and send these display instructions to the projector 112.

[0030] For illustration purposes, Figure 2 The XR environment 200 is shown associated with a single eye 120 of the user, but in other embodiments, another projector 112, waveguide 115, or controller 130 that is completely or partially separate from the XR environment 200 can provide image light to the other eye of the user. In a partially separate system, one or more components can be shared between the waveguides for each eye. In one embodiment, a single waveguide 115 can provide image light to both eyes of the user. Furthermore, in some examples, the waveguide 115 can be one of multiple waveguides of the XR environment 200. In another embodiment where the HMD includes a see-through HMD, the image light can be provided to, for example, one or more transparent or semi-transparent mirrors that can be placed in front of the user's eyes.

[0031] In some embodiments, the projector 112 may include one or more light sources, optical systems, and / or circuits. The projector 112 may generate projection light 155 including at least one two-dimensional image of the virtual content 157 and project the projection light to the first coupling region 150 located on the top surface 270 of the waveguide 115. The image light 155 may propagate along a dimension or axis toward the coupling region 150, for example, as described above with reference to Figure 1 The projector 112 may include one or more array light sources. The techniques and architectures described herein may be applicable to many suitable types of displays, including but not limited to liquid crystal displays (LCDs), liquid crystal on silicon (LCOS), light emitting diodes (LEDs), organic LEDs (OLEDs), micro LEDs (μLEDs), or digital micromirror devices (DMDs).

[0032] In some embodiments, the waveguide 115 may be an optical waveguide that outputs a two-dimensional perceived image 162 of a real-world scene 116 (e.g., scene light for scene objects 117 and scene 118) that is directed toward the user's eye 120. The waveguide 115 may receive the projection light 155 at a first coupling region 150 that includes one or more coupling elements located on a top surface 270 of the waveguide 115 and / or located within the body of the waveguide 115 and that may guide the projection light 155 to a propagation region of the waveguide 115. The coupling elements of the coupling region 150 may be, for example, a diffraction grating, a holographic grating, one or more cascade reflectors, one or more prismatic surface elements, an array of holographic reflectors, a metamaterial surface, or a combination thereof.

[0033] In some embodiments, each of the coupling elements in the coupling region 150 can have substantially the same area in the X-axis and Y-axis dimensions, and can be separated by a distance along the Z-axis (e.g., on the top surface 270 and the bottom surface 280, or on the top surface 270 but separated by an interface layer (not shown), or on the bottom surface 280 and separated by an interface layer, or both embedded in the body of the waveguide 115 but separated by an interface layer). The coupling region 150 can be understood to extend from the top surface 270 to the bottom surface 280. The coupling region 150 can redirect the received projection light 155 into a propagation region of the waveguide 115 formed between the plurality of decoupling elements 260 in the body of the waveguide 115 according to the first grating vector.

[0034] The decoupling element 260A can redirect the projection light 155 from the total internal reflection of the waveguide 115 so that the projection light 155 can be decoupled by the decoupling element 260B. The decoupling element 260A can be part of the top surface 270 of the waveguide 115, fixed to the top surface, or formed in the top surface. The decoupling element 260B can be part of the bottom surface 280 of the waveguide 115, fixed to the bottom surface, or formed in the bottom surface, so that the decoupling element 260A is opposite to the decoupling element 260B, so that the propagation region extends between the decoupling element and the decoupling element. The decoupling elements 260A and 260B can be, for example, diffraction gratings, holographic gratings, holographic reflector arrays, etc., and can form a decoupling region together. In some embodiments, each of the multiple decoupling elements 260A and 260B can have substantially the same area in the X-axis dimension and the Y-axis dimension, and can be separated by a distance along the Z-axis.

[0035] FIG. 3A to FIG. 3F One or more operational examples of disabling one or more image sensors of a computing device corresponding to capturing a real-world content scene obscured by virtual content or abandoning the results of image processing of the real-world content scene obscured by virtual content are shown. Figure 3AAs shown, real-world scene 302 may be captured by one or more cameras 126A and 126B, displayed on display 110, and perceived by user eyes 120. In some embodiments, real-world scene 302 may include real-world content 304 (e.g., an example of a tree). In some embodiments, as shown in FIG. Figure 3B As shown, an XR experience 306 may be instantiated on the XR display device 100, and the XR experience 306 may include virtual content 308 (e.g., an example of a rubber duck) that may be displayed on the display 110 and perceived by the user's eyes 120. For example, in one embodiment, a real-world scene 302 including real-world content 304 (e.g., an example of a tree) and an XR experience 306 including virtual content 308 (e.g., an example of a rubber duck) may be simultaneously displayed on the display 110 and simultaneously perceived by the user's eyes 120. In some embodiments, the real-world scene 302 may include the real-world content 304 (e.g., an example of a tree).

[0036] In some embodiments, the XR experience 306 including the virtual content 308 (e.g., the example of the rubber duck) may include an XR experience in which the virtual content 308 (e.g., the example of the rubber duck) may occupy a majority of the area of ​​the display 110 and the field of view of the user's eyes 120, such that the virtual content 308 (e.g., the example of the rubber duck) may be highlighted relative to the real-world content 304 (e.g., the example of the tree) (e.g., completely occluding or partially occluding the real-world content). For example, in one or more embodiments, the XR experience 306 including the virtual content 308 (e.g., the example of the rubber duck) may include a work-related XR experience in which there may be a large block of empty space or the like displayed on the display 110 and perceived by the user's eyes 120. In another example, an XR experience 306 including virtual content 308 (e.g., the example of a rubber duck) may include a video (e.g., a movie) or a virtual video game on a virtual screen that a user may be watching, in which a real-world scene 302 including real-world content 304 (e.g., the example of a tree) is displayed only in a peripheral area of ​​the display 110, while a larger area of ​​the display 110 is almost entirely occupied by the XR experience 306 including the virtual content 308 (e.g., the example of a rubber duck).

[0037] Thus, in some embodiments, when a user of the XR display device 100 instantiates an XR experience, the XR display device 100 may determine the context of the XR experience, for example, by determining one or more characteristics (e.g., amount of one or more of shadows, occlusions, collisions, diffuse reflections, specular reflections, etc.) associated with a real-world scene 302 including real-world content 304 (e.g., an example of a tree) and an XR experience 306 including virtual content 308 (e.g., an example of a rubber duck). In some embodiments, based on the one or more characteristics, the XR display device 100 may determine that the virtual content 308 (e.g., an example of a rubber duck) is highlighted relative to the real-world content 304 (e.g., an example of a tree) (e.g., completely occluding or partially occluding the real-world content).

[0038] For example, Figure 3C As shown, displayed image 310 shows an XR experience in which virtual content 308 (e.g., an example of a rubber duck) is highlighted relative to real-world content 304 (e.g., an example of a tree) (e.g., completely occluding or partially occluding the real-world content). In some embodiments, in response to determining that virtual content 308 (e.g., an example of a rubber duck) is highlighted relative to real-world content 304 (e.g., an example of a tree) (e.g., completely occluding or partially occluding the real-world content), the XR display device 110 may deactivate one or more of the one or more cameras 126A and 126B because displaying the real-world content 304 (e.g., an example of a tree) in this XR experience may be determined to be unnecessary. Therefore, Figure 3D A redisplayed image 312 is shown in which only virtual content 308 (e.g., the example of a rubber duck) is displayed on the display 110 and perceived by the user eye 120. In one embodiment, the redisplayed image 312 may be generated by the XR display device 100 in response to deactivating one or more of the one or more cameras 126A and 126B, or by simply abandoning image processing of the real-world content 304 (e.g., the example of a tree).

[0039] Figure 3E Another example is depicted in which a displayed image 315 shows an XR experience in which virtual content 316 (e.g., work-related applications, movie screenings, video games, etc.) is highlighted relative to real-world content 304 (e.g., an example of a tree) (e.g., completely occluding or partially occluding the real-world content). Thus, as similarly described above with respect to Figure 3D As discussed, Figure 3FA redisplayed image 318 is shown in which only virtual content 316 (e.g., work-related applications, movie screenings, video games, etc.) is displayed on the display 110 and perceived by the user eye 120. The redisplayed image 316 may be generated by the XR display device 100 in response to deactivating one or more of the one or more cameras 126A and 126B, or by simply abandoning image processing of the real-world content 304 (e.g., the example of the tree).

[0040] Figure 4 A flowchart of a method 400 for deactivating one or more image sensors of a computing device corresponding to the capture of a real-world content scene occluded by virtual content, or forgoing image processing of a real-world content scene occluded by virtual content, according to the presently disclosed techniques is shown. The method 400 may be performed using one or more processing devices (e.g., an XR display device 100), which may include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific IC (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), or one or more other processing devices that may be suitable for processing image data), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0041] Method 400 may begin at block 402, where one or more processing devices (e.g., XR display device 100) display an XR environment on a display of a computing device. Method 400 may then continue at block 404, where one or more processing devices (e.g., XR display device 100) determine a context of the user's XR environment. For example, in some embodiments, determining the context may include determining one or more features associated with virtual content and real-world content scenes that are included within the displayed XR environment. Method 400 may then continue at block 406, where one or more processing devices (e.g., XR display device 100) determine that the virtual content is highlighted relative to the real-world content scene based on the one or more features.

[0042] Then, method 400 may end at box 408, in which one or more processing devices (e.g., XR display device 100) abandon image processing of the real-world content scene obscured by the virtual content in response to determining that the virtual content is highlighted relative to the real-world content scene. For example, in one embodiment, one or more image sensors may be configured to capture the real-world content scene. In this way, the present embodiment can reduce energy consumption (e.g., power consumption, thermal impact) of the XR display device 100 by dynamically abandoning image processing of the real-world content scene obscured by the virtual content and / or deactivating one or more image sensors of the computing device corresponding to the capture of the real-world content scene obscured by the virtual content.

[0043] Figure 5 An example network environment 500 associated with a virtual reality system is shown. The network environment 500 includes a user 501 interacting with a client system 530, a social networking system 560, and a third-party system 570 connected to each other via a network 510. Figure 5 A particular arrangement of user 501, client system 530, social networking system 560, third-party system 570, and network 510 is shown, but the present disclosure contemplates any suitable arrangement of user 501, client system 530, social networking system 560, third-party system 570, and network 510. By way of example and not limitation, two or more of user 501, client system 530, social networking system 560, and third-party system 570 may be directly connected to each other, bypassing network 510. As another example, two or more of client system 530, social networking system 560, and third-party system 570 may be physically or logically co-located with each other, in whole or in part. Furthermore, although Figure 5A particular number of users 501, client systems 530, social networking systems 560, third-party systems 570, and networks 510 are shown, but the present disclosure contemplates any suitable number of client systems 530, social networking systems 560, third-party systems 570, and networks 510. By way of example and not limitation, network environment 500 may include a plurality of users 501, a plurality of client systems 530, a plurality of social networking systems 560, a plurality of third-party systems 570, and a plurality of networks 510.

[0044] The present disclosure contemplates any suitable network 510. By way of example and not limitation, one or more portions of network 510 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 510 may include one or more networks 510. Link 550 may connect client system 530, social networking system 560, and third-party system 570 to communication network 510, or may connect the client system, the social networking system, and the third-party system to each other. The present disclosure contemplates any suitable link 550. In some embodiments, one or more links 550 include one or more wired (e.g., Digital Subscriber Line (DSL) or DataOver Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In some embodiments, one or more links 550 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 550, or a combination of two or more such links 550. The multiple links 550 do not have to be the same throughout the network environment 500. In one or more aspects, the one or more first links 550 can be different from the one or more second links 550 .

[0045] In some embodiments, the client system 530 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more of these components, and is capable of performing appropriate functions implemented or supported by the client system 530. By way of example and not limitation, the client system 530 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a global positioning system (GPS) device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a virtual reality head-mounted viewer and controller, other suitable electronic devices, or any suitable combination thereof. The present disclosure contemplates any suitable client system 530. The client system 530 may enable network users at the client system 530 to access the network 510. The client system 530 may enable its users to communicate with other users at other client systems 530. The client system 530 may generate a virtual reality environment for users to interact with content.

[0046] In some embodiments, the client system 530 may include a virtual reality (or augmented reality) head mounted view 532, and one or more virtual reality input devices 534 (e.g., virtual reality controllers). A user at the client system 530 may wear the virtual reality head mounted view 532 and use the one or more virtual reality input devices to interact with a virtual reality environment 536 generated by the virtual reality head mounted view 532. Although not shown, the client system 530 may also include a separate processing computer, and / or any other components of the virtual reality system. The virtual reality head mounted view 532 may generate a virtual reality environment 536, which may include system content 538 (including but not limited to an operating system), such as software or firmware updates, and the virtual reality environment may also include third-party content 540, such as content from an application or content dynamically downloaded from the Internet (e.g., web content). The virtual reality head mounted view 532 may include one or more sensors 542 (e.g., accelerometers, gyroscopes, magnetometers) to generate sensor data that tracks the position of the head mounted view 532. The head mounted viewer 532 may also include an eye tracker for tracking the position of the user's eyes or the direction in which they are viewed. The client system may use data from the one or more sensors 542 to determine the speed, orientation, and gravity of the head mounted viewer.

[0047] The one or more virtual reality input devices 534 may include one or more sensors 544 (e.g., accelerometers, gyroscopes, magnetometers, and touch sensors) to generate sensor data that tracks the position of the input device 534 and the position of the user's fingers. The client system 530 may use outside-in tracking, in which a tracking camera (not shown) is placed outside of the virtual reality headset 532 and within the line of sight of the virtual reality headset 532. In outside-in tracking, the tracking camera may track the position of the virtual reality headset 532 (e.g., by tracking one or more infrared LED markers on the virtual reality headset 532). Alternatively or additionally, the client system 530 may utilize inside-out tracking, in which a tracking camera (not shown) may be placed on the virtual reality headset 532 or inside the virtual reality headset 532 itself. In inside-out tracking, the tracking camera may capture images of the real world around the tracking camera, and may use the changing perspective of the real world to determine the position of the tracking camera itself in space.

[0048] The third-party content 540 may include a web browser (e.g., MICROSOFT INTERNET EXPLORER, GOOGLE CHROME, or MOZILLA FIREFOX), and may have one or more attachments, plug-ins, or other extensions (e.g., a TOOLBAR or YAHOOTOOLBAR). A user at the client system 530 may enter a Uniform Resource Locator (URL) or other address that directs the web browser to a particular server (e.g., server 562 or a server associated with the third-party system 570), and the web browser may generate a Hyper Text Transfer Protocol (HTTP) request and transmit the HTTP request to the server. The server may accept the HTTP request and transmit one or more Hyper Text Markup Language (HTML) files to the client system 530 in response to the HTTP request. The client system 530 may render a web interface (e.g., a web page) based on the HTML file from the server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not limitation, a web page interface may be rendered according to an HTML file, an Extensible Hyper Text Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on specific needs. These interfaces may also execute scripts, such as but not limited to those written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, and combinations of markup languages ​​and scripts (e.g., asynchronous JAVASCRIPT and XML (AJAX)), etc. In this document, where appropriate, references to a web page interface include one or more corresponding source files (which a browser may use to render the web interface), and vice versa.

[0049] In some embodiments, social networking system 560 may be a network addressable computing system that may host an online social network. Social networking system 560 may generate, store, receive, and transmit social networking data, such as user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 560 may be accessed directly by other components in network environment 500 or via network 510. By way of example and not limitation, client system 530 may access social networking system 560 directly or via network 510 using a web browser in third-party content 540, or a local application associated with social networking system 560 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In some embodiments, social networking system 560 may include one or more servers 562. Each server 562 may be a single server, or a distributed server across multiple computers or multiple data centers. Server 562 can be various types of servers, such as, but not limited to, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof.

[0050] In some embodiments, each server 562 may include hardware, software, or embedded logic components, or a combination of two or more of these components, for performing appropriate functions implemented or supported by the server 562. In some embodiments, the social networking system 560 may include one or more data repositories 564. The data repository 564 can be used to store various types of information. In some embodiments, the information stored in the data repository 564 can be organized according to a specific data structure. In some embodiments, each data repository 564 can be a relational database, a columnar database, an associative database, or other suitable database. Although the present disclosure describes or illustrates a specific type of database, the present disclosure contemplates any suitable type of database. Some embodiments may provide such an interface: the interface enables the client system 530, the social networking system 560, or the third-party system 570 to manage, retrieve, modify, add, or delete the information stored in the data repository 564.

[0051] In some embodiments, the social networking system 560 may store one or more social graphs in one or more data repositories 564. In some embodiments, the social graph may include multiple nodes, which may include multiple user nodes (each user node corresponds to a specific user) or multiple concept nodes (each concept node corresponds to a specific concept) and multiple edges connecting these nodes. The social networking system 560 may provide users of the online social network with the ability to communicate and interact with other users. In some embodiments, users may join an online social network through the social networking system 560, and may subsequently add connections (e.g., relationships) with multiple other users in the social networking system 560 with whom they want to connect. As used herein, the term "friend" may refer to any other user in the social networking system 560 with whom the user has formed a connection, association, or relationship through the social networking system 560.

[0052] In some embodiments, social networking system 560 may provide users with the ability to take actions on various types of items or objects supported by social networking system 560. By way of example and not limitation, these items and objects may include: groups or social networks to which users of social networking system 560 may belong, events or calendar entries that users may be interested in, computer-based applications that users may use, transactions that allow users to buy or sell items through the service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social networking system 560, or with anything that can be represented by an external system of third-party system 570 that is separate from social networking system 560 and coupled to social networking system 560 via network 510.

[0053] In some embodiments, the social networking system 560 may be able to link various entities. As an example and not limitation, the social networking system 560 may enable multiple users to interact with each other and receive content from the third-party system 570 or other entities, or may allow users to interact with these entities through an application programming interface (API) or other communication channel. In some embodiments, the third-party system 570 may include one or more types of servers, one or more data repositories, one or more interfaces (including but not limited to, APIs), one or more network services, one or more content sources, one or more networks, or any other suitable components (e.g., components that can communicate with servers). The third-party system 570 may be operated by an entity different from the entity that runs the social networking system 560. However, in some embodiments, the social networking system 560 and the third-party system 570 may operate in conjunction with each other to provide social networking services to users of the social networking system 560 or the third-party system 570. In this sense, the social networking system 560 may provide a platform or backbone network that other systems (e.g., the third-party system 570) can use to provide social networking services and functions to users on the Internet.

[0054] In some embodiments, third-party system 570 may include a third-party content object provider. The third-party content object provider may include one or more content object sources that may be transmitted to client system 530. By way of example and not limitation, content objects may include information about things or activities of interest to the user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0055] In some embodiments, the social networking system 560 also includes user-generated content objects that can enhance the user's interaction with the social networking system 560. User-generated content can include any content that a user can add, upload, send, or "post" to the social networking system 560. By way of example and not limitation, a user transmits a post from a client system 530 to the social networking system 560. A post can include data such as a status update or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the social networking system 560 by a third party through a "communication channel" (e.g., a news feed or stream). In some embodiments, the social networking system 560 can include various servers, subsystems, programs, modules, logs, and data repositories. In some embodiments, the social networking system 560 can include one or more of the following: a network server, an operation logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an advertisement targeting module, a user interface module, a user profile repository, a contact repository, a third-party content repository, or a location repository. Social-networking system 560 may also include suitable components, such as network interfaces, security mechanisms, load balancers, failover servers, management and network operations consoles, other suitable components, or any suitable combination thereof.

[0056] In some embodiments, the social networking system 560 may include one or more user profile repositories for storing user profiles. User profiles may include, for example, biometric information, demographic information, behavioral information, social information, or other types of descriptive information (e.g., work experience, educational history, hobbies or preferences, interests, in-laws, or locations). Interest information may include interests related to one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an item about a shoe brand, the category may be the brand, or may be the general category "shoes" or "clothing". A contact repository may be used to store contact information about a user. Contact information may indicate users with similar or common work experience, group membership, hobbies, educational history, or users who are related or share common attributes in any way. Contact information may also include user-defined connections between different users and content (internal content and external content). A web server may be used to link the social networking system 560 and one or more client systems 530 or one or more third-party systems 570 via a network 510. The network server may include a mail server or other messaging function for receiving and routing messages between the social networking system 560 and one or more client systems 530. The API request server may allow third-party systems 570 to access information from the social networking system 560 by calling one or more APIs. The operation logger may be used to receive communications from the web server regarding the user's actions on or outside the social networking system 560.

[0057] Along with the operation log, a third-party content object log of the user's exposure to the third-party content object can be maintained. The notification controller can provide information about the content object to the client system 530. The information can be pushed to the client system 530 as a notification, or the information can be extracted from the client system 530 in response to a request received from the client system 530. The authorization server can be used to implement one or more privacy settings of users of the social networking system 560. The privacy settings of the user determine how specific information associated with the user can be shared. The authorization server can allow the user to choose to have or not have their actions recorded by the social networking system 560 or shared with other systems (e.g., third-party system 570), for example, by setting appropriate privacy settings. The third-party content object repository can be used to store content objects received from a third party (e.g., third-party system 570). The location repository can be used to store location information received from the client system 530 associated with the user. The advertising targeting module can combine social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0058] Figure 6An example computer system 600 is shown, which can be used to perform one or more of the aforementioned technologies as currently disclosed herein. In some embodiments, one or more computer systems 600 perform one or more steps of one or more methods described or shown herein. In some embodiments, one or more computer systems 600 provide the functions described or shown herein. In some embodiments, software running on one or more computer systems 600 performs one or more steps of one or more methods described or shown herein, or provides the functions described or shown herein. Certain embodiments include one or more parts of one or more computer systems 600. In this article, references to computer systems may cover computing devices, and vice versa, where appropriate. In addition, references to computer systems may cover one or more computer systems, where appropriate.

[0059] The present disclosure contemplates any suitable number of computer systems 600. The present disclosure contemplates computer systems 600 in any suitable physical form. By way of example and not limitation, computer system 600 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a network of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these systems. Where appropriate, computer system 600 may include one or more computer systems 600; may be single or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may be located in a cloud (which may include one or more cloud components in one or more networks). Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations.

[0060] As an example and not limitation, one or more computer systems 600 may perform one or more steps of one or more methods described or shown herein in real time or in batch processing mode. Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or shown herein at different times or in different locations. In some embodiments, computer system 600 includes processor 602, memory 604, storage device 606, input / output (I / O) interface 608, communication interface 610, and bus 612. Although the present disclosure describes and illustrates a particular computer system with a particular number of particular components in a particular arrangement, the present disclosure contemplates any suitable computer system with any suitable number of any suitable components in any suitable arrangement.

[0061] In some embodiments, the processor 602 includes hardware for executing instructions (e.g., those that constitute a computer program). By way of example and not limitation, to execute instructions, the processor 602 may retrieve (or fetch) these instructions from an internal register, an internal cache, memory 604, or storage device 606; decode and execute these instructions; and then write one or more results to an internal register, an internal cache, memory 604, or storage device 606. In some embodiments, the processor 602 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 602 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 604 or storage device 606, and the instruction cache may speed up the retrieval of these instructions by the processor 602.

[0062] The data in the data cache may be a copy of data in the memory 604 or storage device 606 for operation by instructions executed at the processor 602; may be the result of a previous instruction executed at the processor 602 for access by a subsequent instruction executed at the processor 602 or for writing to the memory 604 or storage device 606; or may be other suitable data. The data cache may speed up read or write operations performed by the processor 602. The TLB may speed up virtual address translation of the processor 602. In some embodiments, the processor 602 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 602 including any suitable number of any suitable internal registers. Where appropriate, the processor 602 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 602. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0063] In some embodiments, the memory 604 includes a main memory for storing instructions for the processor 602 to execute or data for the processor 602 to operate. As an example and not a limitation, the computer system 600 can load instructions from the storage device 606 or another source (e.g., another computer system 600) to the memory 604. Then, the processor 602 can load these instructions from the memory 604 to an internal register or an internal cache memory. In order to execute these instructions, the processor 602 can retrieve these instructions from the internal register or the internal cache memory and decode them. During or after the execution of these instructions, the processor 602 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache memory. Then, the processor 602 can write one or more of these results to the memory 604. In some embodiments, the processor 602 only executes instructions in one or more internal registers or internal cache memories or in the memory 604 (rather than the storage device 606 or elsewhere), and only operates data in one or more internal registers or internal cache memories or in the memory 604 (rather than the storage device 606 or elsewhere).

[0064] One or more memory buses (each of which may include an address bus and a data bus) may couple the processor 602 to the memory 604. As described below, the bus 612 may include one or more memory buses. In some embodiments, one or more memory management units (MMUs) are located between the processor 602 and the memory 604 and facilitate access to the memory 604 requested by the processor 602. In some embodiments, the memory 604 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, the memory 604 may include one or more memories 604. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0065] In some embodiments, storage device 606 includes a large capacity storage device for data or instructions. As an example and not limitation, storage device 606 may include a hard disk drive (HDD), a floppy disk drive (FDD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these storage devices. Where appropriate, storage device 606 may include removable media or non-removable (or fixed) media. Where appropriate, storage device 606 may be inside or outside computer system 600. In some embodiments, storage device 606 is a non-volatile, solid-state memory. In some embodiments, storage device 606 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates mass storage device 606 in any suitable physical form. Where appropriate, storage device 606 may include one or more memory control units that facilitate communication between processor 602 and storage device 606. Where appropriate, storage device 606 may include one or more storage devices 606. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable storage device.

[0066] In some embodiments, the I / O interface 608 includes the following hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication between the computer system 600 and one or more I / O devices. Where appropriate, the computer system 600 may include one or more of these I / O devices. One or more of these I / O devices can implement communication between a person and the computer system 600. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more of these I / O devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 608 for the I / O device. Where appropriate, the I / O interface 608 may include one or more devices or software drivers: the one or more devices or software drivers enable the processor 602 to drive one or more of these I / O devices. Where appropriate, I / O interface 608 may include one or more I / O interfaces 608. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0067] In some embodiments, the communication interface 610 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 600 and one or more other computer systems 600 or one or more networks. By way of example and not limitation, the communication interface 610 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired-based network, or may include a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (e.g., a Wi-Fi network). The present disclosure contemplates any suitable network and any suitable communication interface 610 for the network.

[0068] As an example and not limitation, the computer system 600 can communicate with the following network: an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more parts of the Internet, or a combination of two or more of these networks. One or more parts of one or more of these networks can be wired or wireless. As an example, the computer system 600 can communicate with the following network: a wireless PAN (wireless PAN, WPAN) (e.g., Bluetooth (BLUETOOTH) WPAN), a WI-FI network, a WI-MAX network, a cellular phone network (e.g., a global system for mobile communications (Global System for Mobile Communication, GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 600 may include any suitable communication interface 610 for any of these networks. Where appropriate, the communication interface 610 may include one or more communication interfaces 610. Although the present disclosure describes and illustrates a specific communication interface, the present disclosure contemplates any suitable communication interface.

[0069] In some embodiments, the bus 612 includes hardware, software, or both that couples multiple components of the computer system 600 to each other. By way of example and not limitation, the bus 612 may include: an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these busses. Where appropriate, bus 612 may include one or more buses 612. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0070] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other integrated circuits (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (hybrid hard drives, HHDs), optical disks, optical disc drives (optical disc drives, ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (solid-state drives, SSDs), RAM drives, secure digital cards or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these storage media. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0071] As used herein, "or" is open-ended and non-exclusive unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, "and" is joint and several, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0072] The scope of the present disclosure covers all changes, substitutions, variations, transformations and modifications of the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific parts, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the parts, elements, features, functions, operations, or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references to devices or systems that are suitable for, arranged to, capable of, configured to, enabled to, capable of operating, or operable to perform a specific function, or components in a device or system in the appended claims include the device, system, component, regardless of whether the device, system, component, or the specific function is activated, turned on, or unlocked, as long as the device, system, or component is so adapted to, arranged to, capable of, configured to, enabled to, capable of operating, or operable. Additionally, while this disclosure describes or illustrates certain embodiments as providing particular advantages, certain embodiments may not provide these advantages, or may provide some or all of these advantages.

Claims

1. A method comprising, by a computing device: Displaying an extended reality (XR) environment on a display of the computing device; Determining a context of the XR environment, wherein: Determining the context includes determining one or more characteristics associated with virtual content and real-world content scenes that are included within the displayed XR environment; determining, based on the one or more characteristics, that the virtual content is highlighted relative to the real-world content scene; as well as In response to determining that the virtual content is highlighted relative to the real-world content scene, image processing of the real-world content scene is abandoned.

2. The method according to claim 1, wherein: Determining the one or more characteristics includes determining an amount of one or more of shadows, occlusions, collisions, diffuse reflections, or specular reflections associated with the virtual content and the real-world content scene.

3. The method according to claim 1, wherein: Determining that the virtual content is highlighted relative to the real-world content scene based on the one or more characteristics includes determining that the virtual content obscures a portion of the real-world content scene.

4. The method according to claim 1, further comprising: In response to determining that the virtual content is highlighted relative to the real-world content scene, one or more image sensors configured to capture the real-world content scene are deactivated.

5. The method according to claim 4, wherein: Deactivating the one or more image sensors includes deactivating at least one of the one or more image sensors, the at least one of the one or more image sensors being configured to capture an occluded portion of the real-world content scene.

6. The method according to claim 1, wherein: Determining that the virtual content is highlighted relative to the real-world content scene based on the one or more features includes determining that the virtual content completely obscures the real-world content scene.

7. The method according to claim 6, wherein: Abandoning image processing of the real-world content scene further comprises: When the virtual content completely blocks the real-world content scene, abandoning image processing of the real-world content scene; and When the virtual content does not completely block the real-world content scene, image processing of the real-world content scene is performed.

8. The method according to claim 1, further comprising: In response to determining that the virtual content is highlighted relative to the real-world content scene, redisplaying the XR environment without the real-world content scene on the display.

9. A computing device comprising: at least one display; one or more image sensors; one or more non-transitory computer-readable storage media comprising instructions; and one or more processors coupled to the at least one display, the one or more image sensors, and the non-transitory computer-readable storage medium, the one or more processors configured to execute instructions for: Displaying an extended reality (XR) environment on the at least one display; determining a context of the XR environment, wherein determining the context comprises determining one or more characteristics associated with virtual content and real-world content scenes included within the displayed XR environment; determining, based on the one or more characteristics, that the virtual content is highlighted relative to the real-world content scene; and In response to determining that the virtual content is highlighted relative to the real-world content scene, image processing of the real-world content scene is abandoned.

10. The computing device of claim 9, wherein: The instructions for determining the one or more characteristics associated with the virtual content and the real-world content scene further include instructions for determining an amount of one or more of shadows, occlusions, collisions, diffuse reflections, or specular reflections associated with the virtual content and the real-world content scene.

11. The computing device of claim 9, wherein: The instructions for determining that the virtual content is highlighted relative to the real-world content scene based on the one or more features further include instructions for determining that the virtual content obscures a portion of the real-world content scene.

12. The computing device of claim 9, wherein: The instructions further include instructions for: In response to determining that the virtual content is highlighted relative to the real-world content scene, one or more image sensors configured to capture the real-world content scene are deactivated.

13. The computing device of claim 12, wherein: The instructions for deactivating the one or more image sensors further include instructions for deactivating at least one of the one or more image sensors, the at least one of the one or more image sensors being configured to capture an occluded portion of the real-world content scene.

14. The computing device of claim 9, wherein: The instructions for determining that the virtual content is highlighted relative to the real-world content scene based on the one or more features further include instructions for determining that the virtual content completely obscures the real-world content scene.

15. The computing device of claim 14, wherein: The instructions for abandoning image processing of the real-world content scene further include instructions for: When the virtual content completely blocks the real-world content scene, abandoning image processing of the real-world content scene; as well as When the virtual content does not completely block the real-world content scene, image processing of the real-world content scene is performed.

16. The computing device of claim 9, wherein: The instructions further include instructions for: In response to determining that the virtual content is highlighted relative to the real-world content scene, redisplaying the XR environment without the real-world content scene on the display.

17. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing device, cause the computing device to: Displaying an extended reality (XR) environment on a display of the computing device; Determining a context of the XR environment, wherein: Determining the context includes determining one or more characteristics associated with virtual content and real-world content scenes that are included within the displayed XR environment; determining, based on the one or more characteristics, that the virtual content is highlighted relative to the real-world content scene; as well as In response to determining that the virtual content is highlighted relative to the real-world content scene, one or more image sensors of the computing device are deactivated, the one or more image sensors being configured to capture the real-world content scene.

18. The non-transitory computer readable medium of claim 17, wherein: The instructions for determining that the virtual content is highlighted relative to the real-world content scene based on the one or more features further include instructions for determining that the virtual content obscures a portion of the real-world content scene.

19. The non-transitory computer readable medium of claim 17, wherein: The instructions for determining that the virtual content is highlighted relative to the real-world content scene based on the one or more features further include instructions for determining that the virtual content completely obscures the real-world content scene.

20. The non-transitory computer readable medium of claim 19, wherein: The instructions for abandoning image processing of the real-world content scene further include instructions for: When the virtual content completely blocks the real-world content scene, abandoning image processing of the real-world content scene; as well as When the virtual content does not completely block the real-world content scene, image processing of the real-world content scene is performed.

Images