Dynamic browser stage

Abstract

The present disclosure relates to systems and methods for displaying three-dimensional (3D) content in a spatial (3D) environment. The systems and methods may include receiving a request from a network domain to display 3D content of a particular size at a location within a spatial 3D environment, identifying whether placement is within an authorized portion of the spatial 3D environment, expanding the authorized portion of the 3D spatial environment to display the 3D content based on a user's authorization to resize the authorized portion, and displaying the 3D content.

Description

[0001] This application is a divisional application of the Chinese patent application "Dynamic Browser Stage" with application number 202080084100.0 (application date is December 2, 2020). Technical Field

[0002] The present disclosure generally relates to systems and methods that facilitate an interactive virtual or augmented reality environment for one or more users. Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality," "augmented reality," or "mixed reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in such a way that they appear to be real or that they can be perceived as real. Virtual reality or "VR" scenarios generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality or "AR" scenarios generally involve digital or virtual image information being presented as an enhancement to the visualization of the actual world around the user; and mixed reality or "MR" involves merging the real and virtual worlds to produce new environments in which physical and virtual objects coexist and interact in real time. In reality, the human visual perception system is very complex, and it is challenging to produce VR, AR, or MR technologies that are rich in presentation that promotes a comfortable and natural feeling of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention

[0004] Embodiments of the present disclosure relate to devices, systems, and methods for facilitating virtual or augmented reality interactions for one or more users.

[0005] Further details of features, objects, and advantages of the present disclosure are described below in the detailed description, drawings, and claims.The foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the scope of the present disclosure.

[0006] In some configurations, a display system for displaying virtual content in a three-dimensional (3D) spatial environment may include: a head-mounted display configured to present the virtual content to an eye of a user of the display system; and circuitry in communication with the head-mounted display. The circuitry may be configured to: receive a request to access network-based 3D content; identify parameters associated with the network-based 3D content, including at least one of: a location in the user's 3D spatial environment at which to display the network-based 3D content, an orientation of the 3D content, or a size of the 3D content; determine, based on the parameters, whether the 3D content can be displayed in an authorized portion of the user's 3D spatial environment; and, in response to determining that the 3D content cannot be displayed in the authorized portion of the 3D spatial environment, resize the authorized portion to allow the 3D content to be displayed in the resized authorized portion.

[0007] In some configurations, a display system for displaying virtual content in a three-dimensional (3D) spatial environment may include: a head-mounted display configured to present the virtual content to an eye of a user of the display system; and circuitry in communication with the head-mounted display. The circuitry may be configured to: receive a request to access content; display the content in a first orientation in an authorized portion of the user's 3D spatial environment; receive a request to display the content in a second orientation in the user's 3D spatial environment; determine whether the content can be displayed in the second orientation in the authorized portion of the user's 3D spatial environment; and, in response to determining that the content cannot be displayed in the second orientation in the authorized portion of the 3D spatial environment, resize the authorized portion to allow display of the content in the second orientation within the resized authorized portion.

[0008] In some configurations, a display system for displaying virtual content in a three-dimensional (3D) spatial environment may include: a head-mounted display configured to present the virtual content to an eye of a user of the display system; and circuitry in communication with the head-mounted display. The circuitry may be configured to: receive a request to access 3D content; identify parameters associated with the 3D content, including at least one of: a location in the user's 3D spatial environment at which to display the 3D content, an orientation of the 3D content, and a size of the 3D content; determine, based on the parameters, whether the 3D content can be displayed in an authorized portion of the user's 3D spatial environment; and, in response to determining that the 3D content cannot be displayed in the authorized portion of the 3D spatial environment, display a representation of the 3D content at the location in the 3D spatial environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Throughout the drawings, reference numerals are reused to indicate corresponding relationships between referenced elements.The following drawings and associated descriptions are provided to illustrate embodiments of the present disclosure and do not limit the scope of the claims.

[0010] The accompanying drawings illustrate the design and practicality of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements of similar structure or function are represented by the same reference numerals throughout the drawings. In order to better understand how to obtain the described and other advantages and objectives of the various embodiments of the present disclosure, a more detailed description of the present disclosure briefly described above will be provided with reference to specific embodiments of the present disclosure shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the present disclosure and therefore should not be considered to limit its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1 Illustration depicting a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person.

[0012] Figure 2 An example of a wearable system that may implement an example waypoint system is schematically shown.

[0013] Figure 3 Aspects of a method of simulating a three-dimensional image using multiple depth planes are schematically illustrated.

[0014] Figure 4 Schematically shows an example of a waveguide stack for outputting image information to a user.

[0015] Figure 5 An example output beam that may be output by a waveguide is shown.

[0016] Figure 6 is a schematic diagram showing an optical system for generating a multi-focal volumetric display, image, or light field, which includes a waveguide device, an optical coupler subsystem that optically couples light into or out of the waveguide device, and a control subsystem.

[0017] Figure 7 An augmented reality environment for deconstructing 2D content to be displayed within a user's 3D environment is shown in accordance with some implementations.

[0018] Figure 8 Shown is an example mapping of elements of 2D content to a user's 3D environment in accordance with some implementations.

[0019] Figure 9A Shown is an example browser tile displayed within a bounded volume in 3D space.

[0020] Figure 9B Shown is an example browser block with 3D web content in a bounded volume of 3D space.

[0021] Figure 10AAn example case is shown where the 3D web content is larger than the bounded volume of the 3D space.

[0022] Figure 10B An example case is shown where the center of the 3D web content does not intersect the fixed volume of the 3D space.

[0023] Figure 11 An example content stage or volume is shown.

[0024] Figure 12A1 and Figure 12A2 An example of placing 3D content in a 3D volume is shown.

[0025] Figure 12B1 and Figure 12B2 An example of resizing a 3D volume using a fixed range is shown.

[0026] Figure 13A is a flow chart of an exemplary volume resizing process.

[0027] Figure 13B An exemplary volume resizing is shown.

[0028] Figure 14A is a flow chart of an exemplary page rotation process.

[0029] Figure 14B1 and Figure 14B2 An exemplary page rotation is shown.

[0030] Figure 15A is a flow chart of an exemplary content processing process.

[0031] Figure 15B Two examples of content processing are shown. DETAILED DESCRIPTION

[0032] A. introduce

[0033] Virtual and augmented reality environments are generated by computers using data describing the environment. This data can describe, for example, various objects that a user can perceive and interact with. Examples of these objects include objects rendered and displayed for the user to see, audio played for the user to hear, and tactile feedback for the user to feel. Users can perceive and interact with virtual and augmented reality environments through various visual, auditory, and tactile means.

[0034] Virtual or augmented reality (AR) systems can be useful for many applications, including scientific visualization, medical and military training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. In contrast to virtual reality, augmented reality involves one or more virtual objects that are associated with real objects in the physical world. This experience greatly enhances the user experience and enjoyment of AR systems and opens the door to a variety of applications that allow users to experience real and virtual objects simultaneously.

[0035] However, there are significant challenges in providing such a system. To provide a realistic augmented reality experience to the user, the AR system should always be aware of the user's physical environment so that the position of virtual objects can be correctly related to real objects. Furthermore, the AR system should correctly know how to position virtual objects relative to the user's head, body, etc. This requires always having a broad understanding of the user's position relative to the world. In addition, these functions should be performed advantageously so that costs (e.g., energy consumption, etc.) remain low while maintaining speed and performance.

[0036] Therefore, there is a need to improve the system to provide users with realistic augmented reality experience.

[0037] B. Example of 3D display for wearable systems

[0038] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present a 2D or 3D virtual image to a user. The image can be a still image, a frame of a video, or a video, or a combination thereof. At least a portion of the wearable system can be implemented on a wearable device, which can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be a head-mounted device (HMD), which can be used interchangeably as an AR device (ARD). Further, for the purposes of this disclosure, the term "AR" and the term "MR" are used interchangeably.

[0039] Figure 1 Illustration depicting a mixed reality scene with certain virtual reality objects and certain physical objects viewed by a person. Figure 1 , an MR scene 100 is depicted in which a user of MR technology sees a real-world park-like setting 110 featuring people, trees, and buildings in the background, as well as a concrete platform 120. In addition to these items, the user of MR technology also perceives that he "sees" a robotic statue 130 standing on the real-world platform 120 and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic representation of a bumblebee, even though these elements do not exist in the real world.

[0040] In order for a 3D display to produce a realistic sense of depth, and more specifically, a simulated sense of surface depth, it may be desirable for each point in the display's field of view to generate an accommodation response that corresponds to its virtual depth. If the accommodation response to a displayed point does not correspond to that point's virtual depth, as determined by binocular depth cues for convergence and stereopsis, the human eye may experience accommodation conflict, leading to unstable imaging, unwanted eye strain, headaches, and, in the absence of accommodation information, a near-total lack of surface depth.

[0041] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images can be different for each depth plane (e.g., providing a slightly different presentation of a scene or object) and can be focused separately by the viewer's eyes, thereby facilitating providing depth cues to the user based on the accommodation of the eyes required to focus on different image features of the scene located at different depth planes or based on observing that different image features at different depth planes are out of focus. As discussed elsewhere herein, such depth cues provide a credible perception of depth.

[0042] Figure 2 An example of a wearable system 200 that can be configured to provide an AR / VR / MR scene and that can include the example waypoint system described herein is shown. Wearable system 200 may also be referred to as AR system 200. Wearable system 200 includes a display 220 and various mechanical and electronic modules and systems that support the functionality of display 220. Display 220 can be coupled to a frame 230 that can be worn by a user, wearer, or viewer 210. Display 220 can be positioned in front of the eyes of user 210. Display 220 can present AR / VR / MR content to the user. Display 220 may include a head-mounted display worn on the user's head. In some embodiments, a speaker 240 is coupled to frame 230 and positioned near the user's ear canal (in some embodiments, another speaker, not shown, can be positioned near the user's other ear canal to provide stereo sound / modifiable sound control). Display 220 may include an audio sensor (e.g., a microphone) 232 for detecting an audio stream from the environment and capturing ambient sound. One or more other audio sensors, not shown, may be positioned to provide stereo sound reception. Stereo sound reception can be used to determine the location of a sound source. Wearable system 200 can perform sound or voice recognition on the audio stream.

[0043] The wearable system 200 may include an externally facing imaging system 464 (in Figure 4 ), which observes the world in the environment surrounding the user. The wearable system 200 may also include an internally facing imaging system 462 (shown in Figure 4 210 ), which can be used to track the user's eye movements. The inward-facing imaging system can track the movement of one eye or both eyes. Inward-facing imaging system 462 can be attached to frame 230 and can be in electrical communication with processing module 260 or 270, which can process image information acquired by the inward-facing imaging system to determine, for example, pupil diameter or orientation of the user's 210 eye, eye movement, or eye posture.

[0044] As an example, the wearable system 200 can acquire an image of the user's gesture using the external-facing imaging system 464 or the internal-facing imaging system 462. The image can be a still image, a frame of a video, or a video.

[0045] The display 220 can be operably coupled to a local data processing module 260, for example, via a wired or wireless connection 250, and the local data processing module 260 can be mounted in various configurations, for example, fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-connected configuration).

[0046] The local processing and data module 260 may include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in processing, caching, and storing data. The data may include: a) data captured from a sensor (which may, for example, be operatively coupled to the frame 230 or otherwise attached to the user 210), such as an image capture device (e.g., a camera in an interior-facing imaging system or an exterior-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a radio, or a gyroscope; or b) data acquired or processed using the remote processing module 270 or remote data repository 280, which may be used for communication to the display 220 after such processing or retrieval. The local processing and data module 260 may be operatively coupled to the remote processing module 270 or remote data repository 280, such as via a wired or wireless communication link via communication links 262 or 264, so that these remote modules may serve as resources for the local processing and data module 260. Additionally, remote processing module 270 and remote data repository 280 may be operably coupled to each other.

[0047] Remote processing module 270 may include one or more processors configured to analyze and process data or image information. Remote data repository 280 may include a digital data storage facility that may be accessible via the Internet or other network configuration in a "cloud" resource configuration. Data may be stored and calculations performed in the local processing and data module, allowing for fully autonomous use of the remote module.

[0048] The human visual system is complex, and providing a realistic perception of depth is challenging. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. The convergence movement of the two eyes relative to each other (e.g., the rolling movement of the pupils toward or away from each other to fix the eyes' line of sight on an object) is closely related to the focusing (or "accommodation") of the lenses of the eyes. Under normal circumstances, changing the focus of the lenses of the eyes, or accommodating the eyes to change focus from one object to another at a different distance, will automatically result in a matching change in vergence to the same distance, according to a relationship known as the "accommodation-vergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in accommodation. Display systems that provide a better match between accommodation and vergence can result in a more realistic and comfortable simulation of a three-dimensional image.

[0049] Figure 3 Aspects of a method for simulating a three-dimensional image using multiple depth planes are shown. Figure 3 , objects at different distances from eyes 302 and 304 on the z-axis are accommodated by eyes 302 and 304 to bring those objects into focus. Eyes 302 and 304 assume specific accommodation states to bring objects at different distances along the z-axis into focus. Thus, a specific accommodation state can be considered to be associated with a specific depth plane in depth planes 306, with that specific depth plane having an associated focal length such that when the eye is in the accommodation state for that depth plane, objects or portions of objects in that specific depth plane are in focus. A three-dimensional image can be simulated by providing a different representation of an image to each eye 302, 304 and also by providing a different representation of the image corresponding to each depth plane. Although shown as separate for clarity of illustration, it should be understood that the fields of view of eyes 302 and 304 can overlap, for example, as distance along the z-axis increases. Furthermore, although shown flat for ease of illustration, it will be understood that the contours of the depth planes can be curved in physical space such that all features in the depth planes are in focus when the eyes are in a specific accommodation state. Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Therefore, by providing the eye with a different representation of an image corresponding to each of these limited number of depth planes, a highly believable perceived depth simulation can be achieved.

[0050] C. Waveguide stack components

[0051] Figure 4 An example of a waveguide stack for outputting image information to a user is shown. The wearable system 400 includes a waveguide stack or stacked waveguide assembly 480 that can be used to provide three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 440b. The wearable system 400 may correspond to Figure 2 The wearable system 200, wherein Figure 4 Some parts of the wearable system 200 are shown schematically in more detail. For example, the waveguide assembly 480 can be integrated into Figure 2 in the display 220 .

[0052] Continue to refer Figure 4 The waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between the waveguides. Features 458, 456, 454, 452 may be lenses. Features 458, 456, 454, 452 may not be lenses. Instead, they may simply be spacers (e.g., a cladding or structure used to form an air gap).

[0053] The waveguides 432b, 434b, 436b, 438b, 440b or the plurality of lenses 458, 456, 454, 452 can be configured to deliver image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level can be associated with a particular depth plane and can be configured to output image information corresponding to that depth plane. An image injection device 420, 422, 424, 426, 428 can be used to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, each of which can be configured to distribute incident light across each respective waveguide for output toward the eye 410. Light exits the output surface of the image injection device 420, 422, 424, 426, 428 and is injected into the corresponding input edge of the waveguide 440b, 438b, 436b, 434b, 432b. A single beam (e.g., a collimated beam) can be injected into each waveguide to output an entire field of replicated collimated beams directed toward the eye 410 at specific angles (and divergences) corresponding to the depth plane associated with the particular waveguide.

[0054] The image injection devices 420, 422, 424, 426, 428 can be separate displays, each of which generates image information for injection into the corresponding waveguides 440b, 438b, 436b, 434b, 432b, respectively. Additionally or alternatively, the image injection devices 420, 422, 424, 426, 428 can be outputs of a single multiplexed display that can deliver image information to each of the image injection devices 420, 422, 424, 426, 428, for example, via one or more light guides (such as fiber optic cables).

[0055] Controller 460 controls the operation of stacked waveguide assembly 480 and image injection devices 420, 422, 424, 426, 428. Controller 460 includes programming (e.g., instructions in a non-transitory computer readable medium) that regulates the timing of image information delivered to waveguides 440b, 438b, 436b, 434b, 432b and provides the image information. Controller 460 may be a single integrated device or a distributed system connected via wired or wireless communication channels. In some embodiments, controller 460 may be a processing module 260 or 270 (in Figure 2 ).

[0056] The waveguides 440b, 438b, 436b, 434b, 432b can be configured to propagate light within each respective waveguide by total internal reflection (TIR). Each of the waveguides 440b, 438b, 436b, 434b, 432b can be planar or have another shape (e.g., curved), having a major top surface and a major bottom surface and an edge extending between these major top and bottom surfaces. In the configuration shown, each of the waveguides 440b, 438b, 436b, 434b, 432b can include light extraction optical elements 440a, 438a, 436a, 434a, 432a that are configured to extract light from the waveguides by redirecting the light propagating within each respective waveguide out of the waveguides to output image information to the eye 410. The extracted light can also be referred to as outcoupled light, and the light extraction optical elements can also be referred to as outcoupling optical elements. The extracted light beam can be output from the waveguide at the location where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical elements (440a, 438a, 436a, 434a, 432a) can be, for example, reflective or diffractive optical features. Although shown as being disposed on the major bottom surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of description and clarity of the drawings, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be disposed on the major top surface or major bottom surface, or can be disposed directly in the volume of the waveguides 440b, 438b, 436b, 434b, 432b. The light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material attached to a transparent substrate to form the waveguides 440b, 438b, 436b, 434b, 432b. The waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic material, and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on a surface of or within the material.

[0057] Continue to refer Figure 4As discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b can be configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye can be configured to deliver collimated light injected into such waveguide 432b to the eye 410. This collimated light can represent the optical infinity focal plane. The subsequent uplink waveguide 434b can be configured to emit collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the subsequent uplink waveguide 434b as coming from a first focal plane that is closer inward from optical infinity toward the eye 410. Similarly, the third uplink waveguide 436b passes its output light through a first lens 452 and a second lens 454 before reaching the eye 410. The combined optical power of the first lens 452 and the second lens 454 can be configured to produce another incremental wavefront curvature so that the eye / brain interprets light from the third waveguide 436b as coming from a second focal plane that is closer inward from optical infinity toward the person, rather than from the next upstream waveguide 434b.

[0058] The other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all the lenses between it and the eye for a collective optical power representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 can be provided at the top of the stack to compensate for the aggregate optical power of the underlying lens stack 458, 456, 454, 452. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optics of the waveguides and the focusing aspects of the lenses can be static (e.g., non-dynamic, or electro-active). Additionally or alternatively, either or both can be dynamic using electro-active features.

[0059] Continue to refer Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to redirect light out of their respective waveguides and output that light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different light extraction optical element configurations, with the light extraction optical elements outputting light with different amounts of divergence depending on the associated depth plane. As discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume features or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, or diffraction gratings. Light extraction optical elements, such as diffraction gratings, are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015, which is incorporated herein by reference in its entirety.

[0060] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency so that only a portion of the light beam is deflected toward the eye 410 by each intersection with the DOE, while the remainder continues to move through the waveguide via total internal reflection. Thus, the light carrying image information can be split into multiple related exit beams that exit the waveguide at multiple locations, and for this particular collimated light beam bouncing within the waveguide, the result is a fairly uniform exit emission pattern toward the eye 304.

[0061] One or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which a microdroplet includes a diffraction pattern in a host medium, and the refractive index of the microdroplet can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplet can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0062] The number and distribution of depth planes, or depth of field, can be dynamically varied based on the pupil size or orientation of the viewer's eye. The depth of field can vary inversely with the viewer's pupil size. As a result, as the size of the viewer's eye's pupil decreases, the depth of field increases, such that a plane that was previously indistinguishable due to the plane's location beyond the eye's depth of focus can become discernible and appear more focused as the pupil size decreases, commensurate with the increase in depth of field. Similarly, as pupil size decreases, the number of spaced-apart depth planes used to present different images to the viewer can be reduced. For example, a viewer may not be able to clearly perceive details in both a first depth plane and a second depth plane at one pupil size without adjusting the accommodation of the eye away from one depth plane and towards another. However, both depth planes may be simultaneously sufficiently in focus for the user at another pupil size without changing the accommodation.

[0063] The display system can change the number of waveguides receiving image information based on a determination of pupil size or orientation, or upon receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eyes cannot distinguish between two depth planes associated with two waveguides, the controller 460 (which can be an embodiment of the local processing and data module 260) can be configured or programmed to stop providing image information to one of those waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for a waveguide is switchable between on and off states, the DOE can be switched to an off state when the waveguide does receive image information.

[0064] It may be desirable to have the exit beam meet the condition that its diameter is smaller than the diameter of the viewer's eye. However, given the variability in the size of the viewer's pupil, meeting this condition can be challenging. By varying the size of the exit beam in response to determining the size of the viewer's pupil, this condition can be met across a wide range of pupil sizes. For example, as the pupil size decreases, the exit beam size may also decrease. A variable aperture can be used to vary the exit beam size.

[0065] Wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of world 470. This portion of world 470 may be referred to as the field of view (FOV) of the world camera, and imaging system 464 is sometimes referred to as a FOV camera. The FOV of the world camera may be the same as or different from the FOV of viewer 210, which encompasses the portion of world 470 perceived by viewer 210 at a given time. For example, in some cases, the FOV of the world camera may be larger than the FOV of viewer 210 of wearable system 400. The total area available for viewing or imaging by the viewer may be referred to as the field of view (FOR). The FOR may include 4π steradians of solid angle around wearable system 400, as the wearer can move their body, head, or eyes to perceive essentially any direction in space. In other contexts, the wearer's movement may be more restricted, and thus, the wearer's FOR may subtend a smaller solid angle. Images obtained from outward-facing imaging system 464 may be used to track gestures made by the user (eg, hand or finger gestures), detect objects in world 470 in front of the user, and the like.

[0066] The wearable system 400 may include an audio sensor 232 (e.g., a microphone) to capture ambient sounds. As described above, one or more additional audio sensors may be positioned to provide stereo sound reception useful for determining the location of a speech source. As another example, the audio sensor 232 may include a directional microphone, which may also provide useful directional information about where the audio source is located. The wearable system 400 may use information from both the external-facing imaging system 464 and the audio sensor 230 to locate the source of speech, or to determine the active speaker at a particular moment, for example. For example, the wearable system 400 may use voice recognition alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror) to determine the speaker's identity. As another example, the wearable system 400 may determine the speaker's location in the environment based on the sound captured from the directional microphone. The wearable system 400 may parse the sound from the speaker's location using a voice recognition algorithm to determine the content of the speech, and use voice recognition techniques to determine the speaker's identity (e.g., name or other demographic information).

[0067] Wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements, such as eye and facial movements. Inward-facing imaging system 466 can be used to capture images of eye 410 to determine the size or orientation of the pupil of eye 304. Inward-facing imaging system 466 can be used to obtain images for determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris recognition). At least one camera can be utilized for each eye to independently and separately determine the pupil size or pose of each eye, allowing the presentation of image information to each eye to be dynamically customized for that eye. The pupil diameter or orientation of only a single eye 410 can be determined (e.g., using only a single camera per pair of eyes) and assumed to be similar for both eyes of the user. Images obtained by inward-facing imaging system 466 can be analyzed to determine the user's eye pose or emotion, which the wearable system 400 can use to decide which audio or visual content should be presented to the user. Additionally or alternatively, the wearable system 400 can use sensors (such as an IMU, accelerometer, gyroscope, etc.) to determine head pose (e.g., head position or head orientation).

[0068] Wearable system 400 may include a user input device 466 through which a user may input commands to controller 460 to interact with wearable system 400. For example, user input device 466 may include a touchpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., acting as a virtual user input device), and the like. A multi-DOF controller may sense some or all of the controller's possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll). A multi-DOF controller that supports translational motion may be referred to as 3DOF, while a multi-DOF controller that supports both translation and rotation may be referred to as 6DOF. A user may use a finger (e.g., a thumb) to press or swipe on a touch-sensitive input device to provide input to wearable system 400 (e.g., to provide user input to a user interface provided by wearable system 400). The user input device 466 can be held by a user's hand during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 in a wired or wireless manner.

[0069] Figure 5An example of an exit beam output by a waveguide is shown. One waveguide is shown, but it should be understood that other waveguides in the waveguide assembly 480 can have similar functions, where the waveguide assembly 480 includes multiple waveguides. Light 520 can be injected into waveguide 432b at its input edge 432c and propagate within the waveguide 432b via TIR. At the point where the light 520 impinges on the DOE 282, a portion of the light exits the waveguide as an exit beam 510. The exit beams 510 are shown as being substantially parallel, but they can also be redirected to propagate to the eye 410 at an angle (e.g., forming diverging exit beams), depending on the depth plane associated with the waveguide 432b. It should be understood that substantially parallel exit beams can be indicative of a waveguide having light extraction optics that couple light out to form an image at a depth plane that appears to be located at a greater distance from the eye 410 (e.g., optical infinity). Other waveguides or other sets of light extraction optics may output more divergent exit beam patterns that would require the eye 410 to accommodate to a closer distance to focus on the retina and would be interpreted by the brain as light originating from a distance closer to the eye 410 than optical infinity.

[0070] Figure 6 is a schematic diagram illustrating an optical system for generating a multi-focal volume display, image, or light field, the optical system including a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system may include a waveguide device, an optical coupler subsystem for optically coupling light to or from the waveguide device, and a control subsystem. The optical system may be used to generate a multi-focal volume, image, or light field. The optical system may include one or more primary planar waveguides 632a (at Figure 6 The planar waveguides 632a may be similar to those in FIG. Figure 4 The optical system may employ a distributed waveguide arrangement to provide a plurality of waveguides along a first axis ( Figure 6The distributed waveguide device may, for example, include a distributed planar waveguide 622b and at least one DOE 622a (shown by a double-dashed line) associated with the distributed planar waveguide 622b. The distributed planar waveguide 622b may be similar or identical to the main planar waveguide 632b in at least some respects, but have a different orientation therefrom. Similarly, the at least one DOE 622a may be similar or identical to the DOE 632a in at least some respects. For example, the distributed planar waveguide 622b and / or the DOE 622a may comprise the same material as the main planar waveguide 632b or the DOE 632a, respectively. Figure 6 The embodiment of the optical display system 600 shown in FIG. 6 can be integrated into a Figure 2 In the wearable system 200 shown in FIG.

[0071] The relayed and exit pupil expanded light may be optically coupled from the distributed waveguide device into one or more main planar waveguides 632b. The main planar waveguides 632b may be arranged along a second axis (preferably orthogonal to the first axis (e.g., Figure 6 The main planar waveguide 632b relays light along the second axis (e.g., the horizontal axis or the X-axis in the view). In particular, the second axis can be a non-orthogonal axis to the first axis. The main planar waveguide 632b expands the effective exit pupil of the light along the second axis (e.g., the X-axis). For example, the distribution planar waveguide 622b can relay and expand light along the vertical axis or the Y-axis and pass the light to the main planar waveguide 632b, which can relay and expand the light along the horizontal axis or the X-axis.

[0072] The optical system can include one or more colored light sources (e.g., red, green, and blue lasers) 610 that can be optically coupled to the proximal end of a single-mode optical fiber 640. The distal end of the optical fiber 640 can pass through or be received by a hollow tube 642 of piezoelectric material. The distal end extends from the tube 642 as a fixed, free-standing flexible cantilever 644. The piezoelectric tube 642 can be associated with four quadrant electrodes (not shown). The electrodes can, for example, be electroplated on the exterior, outer surface, or outer periphery or diameter of the tube 642. A core electrode (not shown) can also be located in the core, center, inner periphery, or inner diameter of the tube 642.

[0073] Drive electronics 650 (e.g., electrically coupled via wiring 660) drive opposing pairs of electrodes to independently bend the piezoelectric tube 642 in two axes. The extended distal end of the optical fiber 644 has a mechanical resonant mode. The resonant frequency can depend on the diameter, length, and material properties of the optical fiber 644. By vibrating the piezoelectric tube 642 close to the first mechanical resonant mode of the optical fiber cantilever 644, the optical fiber cantilever 644 is vibrated and can sweep a large deflection.

[0074] By stimulating resonance in two axes, the tip of the fiber cantilever 644 is bidirectionally scanned over an area that fills a two-dimensional (2D) scan. By modulating the intensity of the light source(s) 610 synchronously with the scanning of the fiber cantilever 644, the light emitted from the fiber cantilever 644 can form an image. A description of this arrangement is provided in U.S. Patent Publication No. 2014 / 0003762, the entire contents of which are incorporated herein by reference.

[0075] The components of the optical coupler subsystem can collimate the light emitted from the scanning fiber cantilever 644. The collimated light can be reflected by the mirror 648 to the narrow distribution planar waveguide 622b, which contains at least one diffractive optical element (DOE) 622a. The collimated light can be transmitted vertically (relative to the optical fiber cantilever 644) along the distribution planar waveguide 622b by TIR. Figure 6 622a and repeatedly intersects the DOE 622a. The DOE 622a preferably has a low diffraction efficiency. This can cause a small portion of the light (e.g., 10%) to be diffracted toward the edge of the larger main planar waveguide 632b at each intersection with the DOE 622a, and a small portion of the light to continue its original trajectory down the length of the distributed planar waveguide 622b via TIR.

[0076] At each intersection with the DOE 622a, additional light can be diffracted toward the entrance of the main waveguide 632b. By dividing the incident light into multiple outcoupling sets, the exit pupil of light can be vertically expanded by the DOE 622a in the distributed planar waveguide 622b. This vertically expanded light coupled out of the distributed planar waveguide 622b can enter the edge of the main planar waveguide 632b.

[0077] Light entering the main waveguide 632b can be moved horizontally (relative to the Figure 6 Because light propagates horizontally along at least a portion of the length of the main waveguide 632b via TIR, it intersects the DOE 632a at multiple points. The DOE 632a can advantageously be designed or configured to have a phase profile (which is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern) to produce deflection and focusing of the light. The DOE 632a can advantageously have a low diffraction efficiency (e.g., 10%) so that only a portion of the light beam is deflected toward the eye 210 at each intersection point of the DOE 632a, while the remaining portion of the light continues to propagate within the waveguide via TIR.

[0078] At each intersection between the propagating light and the DOE 632a, a small portion of the light is diffracted toward the adjacent surface of the main waveguide 632b, allowing the light to escape TIR and be emitted from the surface of the main waveguide 632b. In addition, the radially symmetric diffraction pattern of the DOE 632a can impart a focus level to the diffracted light, shape the optical wavefronts of the individual beams (e.g., impart curvature), and steer the beams at angles that match the designed focus level.

[0079] Thus, these different paths can cause light to be coupled out of the main planar waveguide 632b through the multiple DOEs 632a at different angles, focus levels, or to produce different fill patterns at the exit pupil. Different fill patterns at the exit pupil can be advantageously used to generate a light field display with multiple depth planes. Each layer in the waveguide assembly or set of layers in the stack (e.g., 3 layers) can be used to generate a corresponding color (e.g., red, blue, green). Thus, for example, a first set of three adjacent layers can be used to generate red, blue, and green light, respectively, at a first focal depth. A second set of three adjacent layers can be used to generate red, blue, and green light, respectively, at a second focal depth. Multiple sets can be used to generate full 3D or 4D color image light fields with various focal depths.

[0080] D. Other components of wearable systems

[0081] In many embodiments, in addition to or instead of the components of the wearable system described above, the wearable system may include other components. The wearable system may, for example, include one or more haptic devices or components. The haptic device or component is operable to provide a sense of touch to the user. For example, when touching virtual content (e.g., a virtual object, a virtual tool, other virtual constructs), the haptic device or component may provide a sense of touch of pressure or texture. The sense of touch may replicate the feel of a physical object represented by the virtual object, or may replicate the feel of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some embodiments, the user may wear a haptic device or component (e.g., a user may wear gloves). In some embodiments, the haptic device or component may be held by the user.

[0082] The wearable system may, for example, include one or more physical objects that the user can manipulate to allow input or interaction with the wearable system. These physical objects may be referred to as totems in this article. Some totems may take the form of inanimate objects, such as a piece of metal or plastic, a wall, or a table surface. In some embodiments, the totem may not actually have any physical input structure (e.g., a key, trigger, joystick, trackball, rocker switch). Instead, the totem may only provide a physical surface, and the wearable system may render a user interface so that the user appears to be on one or more surfaces of the totem. For example, the wearable system may render images of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and a virtual trackpad to appear to be on the surface of a thin rectangular aluminum plate used as a totem. The rectangular plate itself does not have any physical keys, trackpads, or sensors. However, the wearable system may detect the user's operation or interaction or touch with the rectangular plate as a selection or input via a virtual keyboard or virtual trackpad. User input device 466 ( Figure 4 ) may be an embodiment of a totem, which may include a trackpad, a touchpad, a trigger, a joystick, a trackball, a rocker or virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with gestures to interact with the wearable system or other users.

[0083] Examples of haptic devices and totems that may be used with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated herein by reference in its entirety.

[0084] E. Webpage deconstruction

[0085] Using virtual reality, augmented reality and / or mixed reality systems (hereinafter collectively referred to as "mixed reality" systems), a three-dimensional environment is provided for displaying content to users. Traditional methods of displaying 2D content within a browser do not perform well when used in a 3D environment. One reason for this is that, with traditional 2D web browsers, the display area of the display device is limited to the screen area of the monitor that is displaying the content. Therefore, traditional browsers are configured to only know how to organize and display content within that monitor display area. In contrast, a 3D environment is not limited to the strict limitations of the monitor display area. Therefore, traditional 2D browsers perform poorly when used in a 3D environment because traditional browsing technology does not have the function or ability to utilize the 3D environment to display content.

[0086] For example, consider a situation where a user is using a mixed reality device and has placed multiple browser windows associated with different physical locations. For example, the user may have a first browser window open in a first room and a second browser window open while in a second room. Since traditional 2D-based browsers are limited to the display of a given monitor area, this means that traditional browsers do not even have the technology to understand the concept of a physical remote window, let alone the ability to handle the situation of having multiple windows open in multiple physical locations, making it impossible for the user to effectively view, navigate to, and use these multiple windows.

[0087] Therefore, there is a need for an improved method of implementing browsing techniques in a 3D environment.

[0088] Embodiments of the present disclosure deconstruct 2D web pages for display in a spatially organized 3D environment. The 2D web page can originate from a web browser of a head-mounted system, a mobile device (e.g., a cell phone), a tablet, a television, an application, etc. In some embodiments, the 2D web page can be received from another application or device (such as a laptop, a desktop computer, an email application with a link to the 2D web page, an electronic message that references or includes a link to the 2D web page, etc.).

[0089] refer to Figure 7, environment 700 represents a physical environment and systems for implementing the processes described below (e.g., deconstructing 2D content from a web page for display on a 3D surface in the user's physical environment 705, or providing authentication for an application, or for providing a modal browser window). The representative physical environment and systems of environment 100 include the user's physical environment 705 as viewed by user 708 through head-mounted system 760. The representative system of environment 100 also includes access to 2D content (e.g., web pages) via a web browser 710 operatively coupled to a network 720. Network 720 can be the Internet, an intranet, a private cloud network, a public cloud network, etc. Web browser 710 is also operatively coupled to processor 770 via network 720. Although processor 770 is shown as an isolated component separate from head-mounted system 760, in alternative embodiments, processor 770 can be integrated with one or more components of head-mounted system 760 and / or can be integrated into other system components within environment 100, such as, for example, network 720 to access computing network 725 and storage device 730. The processor 770 may be configured with software 750 for receiving and processing information, such as video, audio, and content, received from the head-mounted system 760, the local storage device 740, the web browser 710, the computing network 725, and the storage device 730. The software 750 may communicate with the computing network 725 and the storage device 730 via the network 720. The software 750 may be installed on the processor 770, or in another embodiment, the features and functions of the software may be integrated into the processor 770. The processor 770 may also be configured with a local storage device 740 for storing information used by the processor 770 for quick access, rather than relying on information stored remotely on an external storage device near the user 708. In other embodiments, the processor 770 may be integrated within the head-mounted system 760.

[0090] The user's physical environment 705 is the physical environment surrounding the user 708 as the user moves around and views the user's physical environment 705 through the head mounted system 760. For example, referring to Figure 7, the user's physical environment 705 shows a room with two walls (e.g., a main wall 780 and a side wall 784, the main wall and the side wall being relative to the user's view) and a table 788. On the main wall 780, a rectangular surface 782 is depicted with solid black lines to represent a physical surface with physical boundaries (e.g., a painting or window hanging or attached to the wall), which may be a candidate surface for projecting 2D content onto. On the side wall 784, a second rectangular surface 786 is depicted with solid black lines to represent a physical surface with physical boundaries (e.g., a painting or window hanging or attached to the wall). On the table 788, there may be different objects: 1) a virtual roster 790 in which 2D content can be stored and displayed; 2) a horizontal surface 792 depicted with solid black lines to represent a physical surface with physical boundaries onto which 2D content can be projected; and 3) multiple stacks of virtual square surfaces 794 depicted with dashed black lines to represent, for example, stacks of virtual newspapers on which 2D content can be stored and displayed.

[0091] The web browser 710 can also display blog pages from the Internet or within an intranet or private network. In addition, the web browser 710 can also be any technology that displays digital 2D content. 2D content can include, for example, web pages, blogs, digital pictures, videos, news articles, newsletters, or music. 2D content can be stored in a storage device 730 that the user 708 can access via the network 720. In some embodiments, 2D content can also be streaming content, for example, live video feed or live audio feed. The storage device 730 can include, for example, a database, a file system, a persistent memory device, a flash drive, a cache, etc. In some embodiments, the web browser 710 containing 2D content (e.g., a web page) is displayed via the computing network 725.

[0092] The computing network 725 accesses the storage device 730 to retrieve and store 2D content for display in a web page on the web browser 710. In some embodiments, the local storage device 740 can provide 2D content of interest to the user 708. The local storage device 740 may include, for example, a flash drive, a cache, a hard drive, a database, a file system, etc. The information stored in the local storage device 740 may include recently accessed 2D content or content recently displayed in a 3D space. The local storage device 740 allows for improved performance of the system of the environment 100 by locally providing certain content to the software 750 for use in assisting in deconstructing the 2D content for display in a 3D spatial environment (e.g., a 3D surface in the user's physical environment 705).

[0093] The software 750 comprises a software program stored in a non-transitory computer-readable medium that performs the functions of deconstructing 2D content for display within the user's physical environment 705. The software 750 can be executed on a processor 770, which can be locally attached to the user 708, or in some other embodiments, the software 750 and processor 770 can be included within the head-mounted system 760. In some embodiments, portions of the features and functionality of the software 750 can be stored and executed remotely from the user 708 on a computing network 725. For example, in some embodiments, deconstructing the 2D content can occur on the computing network 725, and the results of the deconstruction can be stored in the storage device 730, wherein the inventory of surfaces of the user's local environment for presenting the deconstructed 2D content can occur within the processor 770, wherein the inventory and mapping of the surfaces are stored in the local storage device 740. In one embodiment, the processes of deconstructing the 2D content, inventorying the local surfaces, mapping elements of the 2D content to the local surfaces, and displaying the elements of the 2D content can all occur locally within the processor 770 and the software 750.

[0094] Head-mounted system 760 may be a virtual reality (VR) or augmented reality (AR) head-mounted system that includes a user interface, a user sensing system, an environment sensing system, and a processor (all not shown). Head-mounted system 760 presents an interface to user 708 for interacting with and experiencing the digital world. This interaction may involve the user and the digital world, one or more other users interfaced with environment 100, and objects within the digital and physical worlds.

[0095] The user interface may include receiving 2D content and selecting elements within the 2D content through user input through the user interface. The user interface may be at least one of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device, or a combination thereof. A tactile interface device is a device that allows a human to interact with a computer through physical sensations and movements. Haptics refers to a human-computer interaction technology that includes tactile feedback or other physical sensations to perform actions or processes on a computing device. In some embodiments, the control interface may be a user interface that allows a user to interact with the MR display system, for example by providing user input to the system and the system responding by executing corresponding commands.

[0096] The user sensing system may include one or more sensors 762 operable to detect specific features, characteristics, or information related to the user 708 wearing the head-mounted system 760. For example, in some embodiments, the sensor 762 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical characteristics / measurements of the user 708, such as one or more of the following: pupil constriction / dilation, angular measurement / positioning of each pupil, sphericity, eye shape (because eye shape changes over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that may be used by the head-mounted system 760 to enhance the user's viewing experience.

[0097] The environment sensing system may include one or more sensors 764 for obtaining data from the user's physical environment 705. Objects or information detected by the sensors 764 can be provided to the head-mounted system 760 as input. In some embodiments, the input can represent the user's interaction with the virtual world. For example, a user (e.g., user 708) viewing a virtual keyboard on a desk (e.g., table 788) can gesture with their fingers as if the user were typing on the virtual keyboard. The motion of the moving fingers can be captured by the sensors 764 and provided to the head-mounted system 760 as input, where the input can be used to change the virtual world or create new virtual objects.

[0098] The sensor 764 may include, for example, a generally outward-facing camera or scanner for interpreting scene information, such as by continuously and / or intermittently projecting infrared structured light. The environment sensing system may be used to map one or more elements of the user's physical environment 705 surrounding the user 708 by detecting and registering the local environment (including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions). Thus, in some embodiments, the environment sensing system may include image-based 3D reconstruction software embedded in a local computing system (e.g., processor 770) and operable to digitally reconstruct one or more objects or information detected by the sensor 764.

[0099] In one example embodiment, the environment sensing system provides one or more of the following: motion capture data (including gesture recognition), depth sensing, facial recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors. It should be understood that the environment sensing system may include other components in addition to those discussed above.

[0100] As mentioned above, in some embodiments, the processor 770 can be integrated with other components of the head-mounted system 760, integrated with other components of the system of the environment 100, or can be Figure 7 7. The processor 770 is shown as a standalone device (wearable or separate from the user 708). The processor 770 can be connected to the various components of the head-mounted system 760 through a physical wired connection or through a wireless connection (such as, for example, a mobile network connection (including cellular telephone and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol). The processor 770 may include a memory module, an integrated and / or attached graphics processing unit, a wireless and / or wired Internet connection, and a codec and / or firmware capable of converting data from sources (e.g., the computing network 725, the user sensing system and the environment sensing system from the head-mounted system 760) into image and audio data, where the image / video and audio can be presented to the user 708 via a user interface (not shown).

[0101] Processor 770 handles data processing for the various components of head-mounted system 760 and the exchange of data between head-mounted system 760 and 2D content from web pages displayed or accessed by web browser 710 and computing network 725. For example, processor 770 may be used to buffer and process the data stream between user 708 and computing network 725, thereby enabling a smooth, continuous, and high-fidelity user experience.

[0102] Deconstructing 2D content from a web page into elements and mapping these elements for display on surfaces in a 3D environment can be done in an intelligent and logical manner. A predetermined set of rules can be used to recommend, suggest, or dictate where to place specific types of elements / content identified within the 2D content / web page. For example, a specific type of 2D content element may have one or more content elements that may need to be mapped to a physical or virtual object surface suitable for storing and displaying the one or more elements, while other types of 2D content elements may be a single object, such as a main video or main article within a web page, in which case the single object can be mapped to the surface that makes the most sense for displaying the single object to the user.

[0103] Figure 8 An exemplary mapping of elements of 2D content to a user's 3D environment is shown, according to some embodiments. Environment 800 depicts 2D content (e.g., a web page) displayed or accessed by a web browser 710 and the user's physical environment 705. Dotted lines with arrows depict elements (e.g., specific types of content) from the 2D content (e.g., a web page) that are mapped to and displayed on the user's physical environment 705. Based on web designer hints or predefined browser rules, specific elements from the 2D content are mapped to specific physical or virtual objects in the user's physical environment 705.

[0104] As an example, the 2D content accessed or displayed by the web browser 710 may be a web page with multiple tabs, wherein the current active tab 860 is displayed and the auxiliary tab 850 is currently hidden until selected for display on the web browser 710. Displayed within the active tab 860 is typically a web page. In this particular example, the active tab 860 is displaying a YOUTUBE page that includes a main video 820, user comments 230, and recommended videos 840. As shown in this example Figure 8 , the primary video 820 may be mapped to be displayed on the vertical surface 782, the user comments 230 may be mapped to be displayed on the horizontal surface 792, and the recommended video 840 may be mapped to be displayed on a vertical surface 786 that is different from the vertical surface 782. In addition, the auxiliary tab 850 may be mapped to be displayed on the virtual card case 790 and / or the multi-stack virtual object 794. In some embodiments, specific content within the auxiliary tab 850 may be stored in the multi-stack virtual object 794. In other embodiments, all content residing within the auxiliary tab 850 may be stored and / or displayed on the multi-stack virtual object 794. Similarly, the virtual card case 790 may contain specific content from the auxiliary tab 850, or the virtual card case 790 may contain all content residing within the auxiliary tab 850.

[0105] The vertical surface 782 can be any type of structure that may already be on the main wall 780 of the room (depicted as the user's physical environment 705), such as a window pane or a picture frame. In some embodiments, the vertical surface 782 can be a blank wall, wherein the head-mounted system 760 determines the optimal size of the frame of the vertical surface 782 to vertically fit the user 708 viewing the main video 820. This determination of the size of the vertical surface 782 can be based at least in part on the distance of the user 708 from the main wall 780, the size and dimensions of the main video 820, the quality of the main video 820, the amount of uncovered wall space, and / or the user's posture when viewing the main wall 780. For example, if the quality of the main video 820 is high definition, the size of the vertical surface 782 can be larger because the quality of the main video 820 will not be adversely affected by the vertical surface 782. However, if the video quality of the main video 820 is poor, having a large vertical surface 782 may greatly hinder the video quality, in which case the methods and systems of the present disclosure can resize / redefine the vertical surface 782 to be smaller to minimize the poor video quality of the solid animation (pixilation).

[0106] Similar to vertical surface 782, vertical surface 786 is a vertical surface on an adjacent wall (e.g., side wall 784) in the user's physical environment 705. In some embodiments, side wall 784 and vertical surface 786 may appear to be inclined surfaces on an incline based on the orientation of user 708. Vertical, inclined surfaces on an incline may be a type of surface orientation in addition to vertical and horizontal surfaces. A recommended video 840 from a YOUTUBE webpage may be placed on vertical surface 786 on side wall 784 to allow user 708 to view the recommended video simply by moving their head slightly to the right in this example.

[0107] Virtual Card Box 790 is a virtual object created by head mounted system 760 and displayed to user 708. Virtual Card Box 790 may have the ability for user 708 to bidirectionally cycle through a set of virtual pages. Virtual Card Box 790 may contain an entire webpage, or it may contain a separate article, video, or audio. As shown in this example, Virtual Card Box 790 may contain a portion of the content from auxiliary tab 850, or in some embodiments, Virtual Card Box 790 may contain the entire page of auxiliary tab 850. User 708 may bidirectionally cycle through the content within Virtual Card Box 790 by focusing only on a specific tab within Virtual Card Box 790, and one or more sensors within head mounted system 760 (e.g., sensor 762) detect the eye focus of user 708 and cycle through the tabs within Virtual Card Box 790 accordingly to obtain relevant information for user 708. In some embodiments, the user 708 can select relevant information from the virtual business card box 790 and instruct the head-mounted system 760 to display the relevant information on an available surrounding surface or on another virtual object, such as a virtual display (not shown) in close proximity to the user 708.

[0108] Similar to the virtual Rolodex 790, the multi-stacked virtual object 794 can contain content ranging from complete content from one or more tabs or specific content from various web pages or tabs that the user 708 has bookmarked, saved for future viewing, or has already opened (e.g., an inactive tab). The multi-stacked virtual object 794 is also similar to a stack of newspapers in the real world. Each stack within the multi-stacked virtual object 794 can belong to a specific newspaper article, page, magazine issue, recipe, etc. It will be understood by those skilled in the art that there can be multiple types of virtual objects that achieve the same purpose of providing a surface for placing 2D content elements or content from a 2D content source.

[0109] It will be appreciated by those skilled in the art that the 2D content accessed or displayed by the web browser 710 may be more than just web pages. In some embodiments, the 2D content may be pictures from an album, videos from movies, TV shows, YOUTUBE videos, interactive forms, and the like. In other embodiments, the 2D content may be an e-book, or any electronic means of displaying a book. Finally, in other embodiments, the 2D content may be other types of content not yet described, since 2D content is typically how information is currently presented. If the electronic device can consume 2D content, the 2D content may be used by the head-mounted system 760 to deconstruct the 2D content and display the 2D content in a 3D setting (e.g., AR).

[0110] In some embodiments, mapping the accessed 2D content may include extracting the 2D content (e.g., from a browser) and placing it on the surface (so that the content is no longer in the browser and is only on the surface), and in some embodiments, mapping may include copying the content (e.g., from a browser) and placing it on the surface (so that the content is both in the browser and on the surface).

[0111] F. Web content in bounded volumes

[0112] In some embodiments, the extracted web content can be placed within a bounded volume within the user's 3D spatial environment. Advantageously, restricting the placement of content to a bounded volume can allow the user to better control the placement of content within their environment. In some examples, the content can be placed with reference to a volume surrounding a browser block that can be part of a web browser application. Figure 9A An example browser tile 902 is shown in a volume 904 of 3D space.

[0113] The browser block 902 can display 2D content, such as some or all of a web page, content associated with a web domain, or non-web-based text or graphic information. For example, the browser block 902 may include a graphical interface for accessing content (such as graphics or text). The browser block 902 may include one or more aspects of a web browser, such as multiple windows or tabs, forward and backward interactive features for navigating content previously interacted with by the user, a refresh interactive feature for refreshing currently viewed content, a cancel interactive feature for canceling loaded content, a homepage interactive feature for returning to preset content, a navigation feature for entering a local or network location for content, or a search feature for searching for local or web-based content. In some examples, the browser block 902 may include 2D content displayed in conjunction with one or more aspects of a web browser, such as described above.

[0114] The AR system may allow a user to interact with the browser block 902 and / or the content displayed in the browser block 902. For example, the AR system may receive an instruction to move or manipulate the browser block 902 via a user gesture, hand gesture, or actuation of a user input device. The AR system may respond by moving or manipulating the browser block 902 accordingly. In some examples, the associated user gesture for moving or manipulating the browser block 902 may include hand movements, body positioning, or some combination thereof. In another example, the AR system may receive an instruction to manipulate the content displayed in the browser block 902 or other content associated with the browser block 902 via a user gesture or actuation of a user input device. The AR system may respond by manipulating the content associated with the browser block accordingly, for example, by scrolling the content, clicking to find new content, selecting content, deleting content, extracting content, entering content, or some combination thereof. Manipulation may include interacting with one or more browser features that may be included in the browser block 902. For example, manipulation may include finding content by entering information into an address or search feature, loading or suspending loading content, or other manipulation as a result of interacting with other interactive features.

[0115] The AR system can display browser tile 902 in volume 904 of 3D space. Volume 904 may include a subset of the user's 3D environment. As described below, volume 904 may be static or manipulable. For example, volume 904 may be expanded, reduced, or otherwise transformed via user instruction and / or automatically by the AR system.

[0116] The AR system may display 3D content 906 associated with the 2D content displayed in the browser block 902 . Figure 9B An example browser tile 902 is shown with 3D content 906 in a volume 904. The 3D content 906 may include one or more virtual objects.

[0117] In some embodiments, the 3D content may include one or more prisms, which generally describe a container, area, or volume associated with the mixed reality content, which may contain multiple virtual content items, such as representations of 3D objects. The restricted content in the prism can be controlled or placed in the user's environment by controlling or placing the prism that restricts the content. As used herein, a virtual object can be or include a prism. Various features, uses, and implementations of prisms are described in U.S. Patent Publication No. 2019 / 0197785, published on June 27, 2019, the entire contents of which are incorporated herein by reference.

[0118] The virtual object can be associated with the 2D content displayed in the browser block 902. For example, the 2D content can be a website for a furniture company, and the virtual object can be a 3D virtual representation of the furniture described on the website. In some examples, the virtual object can be displayed near the browser block 902, such as at a location in the 3D volume 904 that is associated with the location of the graphics displayed on the browser block 902. For example, the 2D content can include a 2D representation of a sofa. The 3D content can include a 3D representation of the sofa. In some examples, the AR system can display a 3D representation of the sofa in front of or instead of the 2D representation of the sofa, such as a representation of the sofa (or other 3D content) extending from the browser block 902.

[0119] The placement and size of virtual objects can be changed or modified by the user. For example, the AR system can place (one or more) virtual objects based on a default position within the volume 904 relative to the browser block 902. In some examples, the AR system can apply transformations to one or more virtual objects, such as rotation, scaling, movement, more complex animations, etc., or some combination thereof.

[0120] like Figure 10A and Figure 10B As shown, the display of the virtual object can be limited to the boundaries of the 3D volume 1002. For example, the AR system can receive an instruction, such as an instruction from a user or an executable instruction, to display a portion of the virtual object 906 outside the boundaries of the volume 1002. The AR system can determine not to display the virtual object 906 or to display a portion of the virtual object 906. Additionally or alternatively, as discussed below, the AR system can resize or reorient the 3D volume 1002 to contain the virtual object 906, for example, by adjusting the position, volume, orientation, and / or location of the virtual object 906.

[0121] In one example, if Figure 10A As shown, virtual object 906A can be placed, sized, or oriented so that a portion 1004 of virtual object 906 falls outside the boundaries of 3D volume 1002. If portion 1004 exceeds a threshold volume or threshold percentage of 3D volume 1002 or virtual object 906A, or other threshold associated with virtual object 906A or 3D volume 1002, the AR system can determine not to display the entire virtual object 906A. If portion 1004 is below the threshold, the AR system can determine to display all or part of virtual object 906A. For example, the AR system can display the portion 1006 of virtual object 906 that falls within 3D volume 1002. In another example, the AR system can display the entire virtual object 906A even if it falls outside 3D volume 1002.

[0122] In another example, Figure 10B As shown, the virtual object 906B can be placed, sized, and / or oriented so that the center 1008 of the virtual object 906B falls outside the 3D volume 1002. If the center 1008 of the virtual object 906B falls outside the 3D volume 1002, the AR system can determine not to display the entire virtual object 906B. In another example, the AR system can determine to display a portion of the virtual object 906B that falls within the 3D volume 1002.

[0123] G. Example bounding volumes

[0124] In some examples, the AR system can define a 3D volume of space (also called a stage) in the user's environment. Figure 11 As shown, 3D volume 1102 may include a region of space in which content 1104 may be created, displayed, or manipulated.

[0125] In some examples, volume 1102 can have a default size or shape. The default size can be set to a predetermined size, can be set relative to the size of content 1104, can be set relative to the user's environment or field of view, or can be set based on another factor.

[0126] For example, the default size can be a cuboid with sides of 0.75 meters, 0.8 meters, 0.86 meters, 1 meter, or other sizes. In another example, the default size can be a cuboid with sides of 1000 pixels, 1100 pixels, 1120 pixels, 1200 pixels, or other sizes. Advantageously, setting the default size based on a set of predetermined dimensions can allow for larger content without having to resize volume 1102.

[0127] In another example, the default size can be a percentage larger than content 1104 in at least one dimension. For example, volume 1102 can be 20% wider and 20% taller than content 1104, and the depth (or breadth) of volume 1104 can be equal to the width of volume 1102. In another example, volume 1102 can be 10% wider than content 1104 and the same height as content 1104. In another example, volume 1102 can be 50% wider and 100% taller than content 1104. However, other examples are possible. Advantageously, setting a default size based on the size of content 1104 can allow for smaller and more centralized placement of new content related to content 1104. For example, content 1104 can be a browser block that allows access to a web domain. The web domain can request placement of new content. Defining a default size relative to the size of the web domain can allow for small content related to the web domain to be placed closer to the browser block.

[0128] In some examples, the default size can be relative to the user's field of view so that volume 1102 fills a percentage of the user's field of view (FOV). For example, volume 1102 can be 10%, 25%, 50%, 100%, 150%, or other percentage amount of the FOV.

[0129] The shape of volume 1102 can be any 3D shape, such as a cube, a cuboid, a sphere, an ellipsoid, a pyramid, a cylinder, some combination thereof, or other shapes. In some examples, the shape of volume 1102 can be a complementary shape to content 1104. For example, content 1104 can be a browser block having a cuboid shape. Volume 1102 can then have a cuboid shape. In another example, content 1104 can be a more complex geometric shape, such as a sofa or a person. Volume 1102 can have a shape that outlines the geometry of content 1104. Additionally or alternatively, volume 1102 can be spherical or cuboid with dimensions that encompass content 1104.

[0130] In some examples, volume 1102 can have a default size based on one or more characteristics of content 1104. For example, content 1104 can be a browser block. Volume 1102 can have a size relatively close to that of a browser block because browser blocks are not designed to move without user input. In another example, content 1104 can be a virtual object (e.g., a butterfly) configured to fly around a user's environment. Volume 1102 can have a size that encompasses a large proportion of the user's FOV to allow content 1104 to move around.

[0131] In some examples, the volume may be a static volume. In other examples, the volume may be manipulable or changeable. Figure 12A1 and Figure 12A2 As shown, volume 1203 may expand to include one or more pieces of content 1204 , 1206 . Figure 12A1 A perspective view 1200 is shown, Figure 12A2 A top view 1201 of a changeable volume 1202 is shown. For example, the initial size of volume 1203 can be a default size relative to the browser block. The user can interact with the web domain using the browser block 1204. The web domain can request the AR system to display or otherwise cause the AR system to display related content 1206 within the user's environment. As a result, the AR system can expand volume 1203 in one or more directions to reach (e.g., become) volume 1202B containing related content 1206 and display content 1206, such as Figure 12A2 shown.

[0132] The volume can be extended or resized using the defined extents. Figure 12B1 and Figure 12B2An example extension using ranges is shown. A range can include a defined amount of distance from a piece of content (such as a browser block). In some examples, if a piece of content is resized, one or more ranges around the piece of content can remain the same while the size of the volume around the content is adjusted. For example, the AR system can define the number of pixels in one or more dimensions to add to content 1224 in the user's environment (in Figure 12B1 Shown as 1224A, Figure 12B2 1224B). In some examples, the range may include 100 pixels to the left, 100 pixels to the right, 150 pixels to the bottom, 250 pixels to the left, and / or 200 pixels to the front. However, other sizes or combinations of ranges are possible. The range may be defined by the requestor, the user, or the AR system.

[0133] In another example, the AR system can define a percentage change in one or more dimensions of the volume. For example, the AR system can increase the width by 20% to the left, 10% to the bottom, 50% to the top, or other combinations of dimensions or percentages. Figure 12B1 and Figure 12B2 In the example shown, the AR system utilizes a set of defined ranges 1222 to expand volume 1205A to volume 1205B. For example, the defined range 1222 can be a number of pixels, such as 100 pixels. The AR system can initially determine that the size of volume 1205A corresponds to the volume of content 1224A plus the range on one or more sides of the volume of content 1224A. After resizing the volume of content 1224A to the volume of content 1224B, the AR system can maintain the same range and expand to volume 1205B, which includes the volume of content 1224B plus the range on one or more sides of the volume of content 1224B. In some examples, the content may include a browser block, and the resizing may correspond to resizing the browser block or a web page associated with the browser block.

[0134] As discussed below, the AR system can control the size of volume 1205A based on one or more of user input and third-party requests or other factors. For example, to prevent a third party (such as a network domain) from automatically expanding volume 1205A when displaying new content, the AR system can determine a set of permitted states, which can include user input. Advantageously, this process can allow the user or the AR system to exert greater control over the use of space within the user's environment, while still allowing third parties to recommend optimized sizes for content or volumes.

[0135] H. Exemplary Resizing Authentication Process

[0136] In some examples, the AR system may control the size of the adjustment volume through an authentication process. Figure 13A is a flow diagram illustrating an example resizing process 1300 that may include an authentication process. Figure 13B An example set of steps is shown that may be part of or a result of one or more steps in the resizing process 1300 .

[0137] refer to Figure 13A At block 1302, the AR system may receive a resize request. The resize request may include a request to resize the volume 1324 from an application, a network domain, a third party, or other source, such as Figure 13B As shown. Adjusting the size of volume 1324 can include expanding or contracting volume 1324 in one or more directions from a point within volume 1324 (such as the center of volume 1324, the center of content 1322 within volume 1324, or another point within the user's 3D environment). The resize request can include a percentage or amount of change in one or more dimensions of volume 1324. For example, the amount of change can include an amount in meters, a number of pixels, or other amounts. In some examples, the amount of change can be based on the size or placement of content added within volume 1324. For example, a user can interact with a web domain via a browser application displayed within volume 1324. Based on the user interaction, the web domain can request that 3D content be displayed within the user's 3D environment. The web domain can send a resize request based on the desired size and placement of the 3D content within volume 1324 (or the AR system can automatically initiate a resize request, for example based on the web domain content) so that 3D volume 1324 contains the added content.

[0138] At block 1304, the AR system can determine whether the resizing is authorized. For example, the AR system can determine whether the user authorizes or has authorized the resizing of the volume 1324 to the requested size. The AR system can make this determination by displaying a prompt to the user, checking the authorization status of the request, checking the authorization status of the requesting party, some combination thereof, or another method.

[0139] To determine authorization by displaying a prompt to the user, the AR system may graphically display a graphic or text requesting user input regarding the resize request. For example, Figure 13BAs shown, the AR system may output a graphic 1328 requesting approval to resize volume 1324. Graphic 1328 may include text and / or one or more interactive components for user input. For example, graphic 1328 may include a prompt to the user regarding whether to allow the stage to be resized. Additionally or alternatively, graphic 1328 may include one or more buttons (such as a "yes" button or a "no" button) or other interactive components for accepting user input. Additionally or alternatively, the AR system may display one or more temporary lines 1326 to show the proposed new size of volume 1324. One or more temporary lines 1326 may include arrows, lines, or other distance indicators. Line 1326 may start from the current boundary of content 1322 (or the current boundary of the prism containing the content) or the current boundary of stage 1324 and extend to the boundary of the newly proposed stage 1324. In some examples, the line may be removed once user input is received. Additionally or alternatively to the temporary lines, the AR system can display the boundaries of the suggested volume 1324, for example, by displaying an outline of the boundaries of the suggested volume 1324 or adding volumetric shading to indicate the size of the suggested volume 1324. If resizing is authorized, for example, by the user accepting a resizing request, the AR system can resize the stage 1324. If resizing is not authorized, the AR system may not resize the stage. For example, the AR system can display a stage 1323 having the same or similar volume as the content 1322.

[0140] To determine authorization by checking the authorization status of the request, the AR system can determine whether the user has previously authorized resizing. For example, a user may authorize all or some resizing requests during a particular interactive session. For example, during a web session, the user may have authorized a volume expansion of 100 pixels in each of six directions. However, the AR system may have already expanded the volume by 25 pixels in each of the six directions when placing the content. The AR system can then determine whether the current resizing request falls within the originally authorized 100-pixel limit in each of the six directions. If the resizing falls within the previously authorized limits, the AR system can resize the stage at block 1306. If the AR system finds that the resizing is not within the limits, the AR system can deny the resizing or display a prompt to the user to authorize the resizing. Additionally or alternatively, the AR system can determine authorization status by checking whether the resizing falls within a threshold change (e.g., a default threshold change or a user-defined threshold change). If the threshold change is below a threshold amount, the AR system can authorize the resizing. If the threshold change exceeds the threshold change, the AR system can deny the resize or display a prompt to the user to authorize the resize.

[0141] To determine authorization by checking the authorization status of the requestor, the AR system can determine whether the user has allowed the requestor to resize the volume 1324 generally during a limited time period and / or within specific limitations. For example, the requestor can be a trusted requestor so that the AR system will approve all resize requests. In another example, the requestor can have limited authorization so that the AR system will approve resize requests within set limits on time, volume change, and / or other parameters. For example, if the user has authorized the requestor (such as a network domain) to resize the volume 1324 during a network session, the AR system can resize the volume during the network session. However, if the user navigates away from the network domain, the AR system may not resize the volume or prompt the user to authorize the resize.

[0142] At block 1306, the AR system can resize the stage. Resizing can include one or more processes, such as those described with respect to FIGURES 12A and 12B. Figure 13B , the AR system can use a percentage change, a range, or another method to adjust the size of volume 1324. In some examples, the AR system can not display an outline of volume 1324 during resizing. In some examples, the AR system can display an outline of volume 1324 during resizing.

[0143] At block 1308 , the AR system may determine whether to refresh the stage. The AR system may determine that the stage should be refreshed based on one or more refresh conditions. Refresh conditions may include one or more states of the content, the source of the content (such as a web domain or application), the AR system, resizing authorization, or other relevant conditions. For example, the AR system may determine whether resizing authorization has expired or resizing is no longer relevant, such as when the user is no longer interacting with the content from the requester. In some examples, a refresh may occur when the user leaves the web domain where resizing was requested. For example, a user may switch from a shopping website that requested resizing to display products in the user's environment to a news website. In another example, a refresh may occur when a web session has ended or a new web session has begun. In another example, a refresh may occur when an application is launched or restarted. In another example, the size of the stage may depend on or be bound to the content or application the user is interacting with. If the user launches another application or loads content with its own stage size requirements, the AR system may refresh the stage. If the AR system determines that a refresh has occurred, then at block 1310 , the AR system may reset the volume 1324 . If the AR system has not determined that a refresh has occurred, the AR system may continue to use the current size of the volume 1324 .

[0144] At block 1310, the AR system can set volume 1324 to a previous size. For example, the AR system can set volume 1324 to the size that the AR system appeared to the user before the resize request. In some examples, the previous size can include a default size for volume 1324, such as the size of a browser block or other content plus some range in one or more dimensions. In other examples, the previous size can include a size that is not a default size.

[0145] I. Example Page Rotation

[0146] In some examples, the AR system can resize the volume to accommodate content rotation, such as rotation of a web page within a browser block. Figure 14A An exemplary rotation process 1400 for rotating a web page within a browser block is shown, and FIG. 14B illustrates an example of volumetric rotation resizing of a web page within a browser block.

[0147] refer to Figure 14A , at block 1402, the AR system may receive a rotation request. The rotation request may include a request from a user or a third-party application to rotate one or more components of content displayed in the user's 3D environment. For example, as shown in FIG14B , the AR system may display a web page 1428 within a browser block 1424. Initially, the web page may be displayed in a vertical orientation. The user, a third party, or an application may request that the web page be oriented in a horizontal orientation. In other examples, the AR system may display multiple contents. The AR system may receive a request to reorient or reposition one or more portions of the content within the user's 3D environment.

[0148] Continue to refer Figure 14A , at box 1404, the AR system can determine a new orientation of the content within the user's 3D environment. For example, the AR system can identify the new orientation based on the user's gesture and / or rotation request. Any number of orientations are possible. In some examples, the new orientation can be horizontal, vertical, diagonal, rotated about the x, y, or z axis, etc., or some combination thereof. For example, if the user is looking at content, such as a browser block, the user can request the orientation of the content within the browser block so that the content is facing (e.g., perpendicular to) the user's gaze direction. In another example, as shown in Figure 14B, the content can be a web page 1428 within a browser block 1424, and the user can request that the web page 1428 be reoriented from a vertical orientation 1401 to a horizontal orientation 1403.

[0149] At block 1406, the AR system may determine a new volume size based on the new orientation. Figure 14B1As shown, the AR system can resize volume 1422A to accommodate page 1428 in the new orientation. New volume 1422B can include the same, larger, or smaller length in one or more dimensions. Figure 14B1 and Figure 14B2 In the example shown, the height h and width w of the volumes may remain the same (eg, in volume 1422A and volume 1422B), but the depth d may increase to the height of page 1428 in volume 1422B. However, other resizing is possible.

[0150] At block 1408, the AR system may determine resize authorization. Resize authorization may include one or more steps for determining whether the resize request is authorized by the user or the AR system. For example, resize authorization may include reference to Figure 13A The AR system may further perform one or more steps of the resizing process 1300 described herein. In some examples, the AR system may display a prompt to the user to determine whether resizing or orientation is authorized. If the AR system determines that resizing is authorized, then at block 1410, the AR system may resize the volume. If resizing is not authorized, the AR system may not resize the volume or display the new content orientation.

[0151] At block 1410, the AR system may resize the volume based on the resize authorization. Resizing may include one or more processes, such as those described with respect to FIGURES 12A and 12B. As applied to FIGURE 14B, for example, the AR system may resize volume 1422 using a percentage change, using a range, or another method. In some examples, the AR system may not display an outline of volume 1422A after resizing from volume 1422. In some examples, the AR system may display an outline of volume 1422A while resizing from volume 1422, e.g., for a predetermined period of time and / or until a specific user interaction occurs.

[0152] At block 1412, the AR system may display the content in the resized volume at the new orientation. The content may be displayed at the requested orientation so that the position of the content falls within the resized volume 1422A.

[0153] J. Exemplary Content Processing

[0154] In some examples, the AR system may receive a request to locate or place content in the user's 3D environment, some or all of which falls outside the boundaries of a defined volume of space in which the content is allowed to be placed. Figure 15A An exemplary content processing process 1500 for placing such content is shown, and Figure 15BExamples of content handling when the volume is resized and when the volume is not resized are shown.

[0155] refer to Figure 15A At block 1502, the AR system may receive a content placement request. A content placement request may include a request from an application, a network domain, a third party, or other source to place content 1526 in an area of a user's 3D environment, such as Figure 15B 1522 . The content placement request may include coordinates and / or dimensions of content 1526. The coordinates may reference volume 1522, the user, and / or another origin within the user's 3D environment. In some examples, the request may include placing some or all of content 1526 outside of volume 1522. In some examples, the request may include placing some or all of content 1526 within volume 1522. For example, a user may interact with a web domain via a browser application displayed within volume 1522. Based on the user interaction, the web domain may request that content 1524 be displayed within volume 1522. Additionally or alternatively, the web domain may send a content placement request based on a desired size and placement of content 1524 within volume 1522.

[0156] At block 1506, the AR system may determine whether a portion of the content will fall outside the boundaries of the volume based on the content placement request. Figure 15B As shown in scenario 1501 , the AR system may receive a request to place content 1526 completely outside the boundaries of volume 1522 . However, other scenarios are possible.

[0157] In another example, the AR system may receive a request to place a portion of content 1526 outside the boundaries of volume 1522. In some examples, if the portion exceeds a threshold volume, a threshold percentage of volume 1522, or other threshold associated with content 1524, 1526, and / or 3D volume 1522, the AR system may determine that content 1526 falls outside the boundaries of volume 1522. In some examples, the threshold may include 5% of the volume, 10% of the volume, or any other percentage. In some examples, the threshold may include 10 pixels, 15 pixels, or any other number of pixels. If the AR system determines that the content falls within the boundaries of the volume, then at block 1518, the AR system may display the content within the volume. If the AR system determines that the content falls outside the boundaries of the volume, then at block 1508, the AR system may determine whether to resize the volume.

[0158] At block 1508, the AR system may determine whether to resize the volume. For example, the AR system may determine whether to resize the content based on the status of the resize authorization. The resize authorization may include one or more steps for determining whether the resize request is authorized by the user or the AR system. For example, the resize authorization may include reference to Figure 13A The AR system may further perform one or more steps in the described resizing process 1300. In some examples, the AR system may display a prompt to the user to determine whether resizing or orientation is authorized. If the AR system determines that resizing is authorized, then in block 1516, the AR system may resize the volume. If resizing is not authorized, then in block 1510, the AR system may determine whether to copy the content.

[0159] At block 1516, the AR system may resize the volume. Resizing may include one or more processes, such as those described with respect to FIG. 12A and FIG. 12B . Figure 15B , for example, the AR system can resize volume 1522 to expand to volume 1522A using a percentage change, using a range, or another method. In some examples, the AR system may not display an outline of volume 1522A after resizing from volume 1522. In some examples, the AR system may display an outline of volume 1522A after resizing from volume 1522. Once the AR system resizes the volume, the AR system may display the content in the resized volume 1522A at the requested location, as in Figure 15B As shown in situation 1503.

[0160] At block 1510 , the AR system may determine whether to copy the content. To determine whether a copy should be made and placed, the AR system may determine whether the user has authorized display of a copy of the content 1526 outside the boundaries of the volume 1522 . Authorization for copying may include one or more steps for determining whether copying is authorized by the user or the AR system. In some examples, the AR system may display a prompt to the user to determine whether copying and displaying the content 1526 outside the volume 1522 at the requested placement is authorized. If the user indicates approval of the copying in response to the prompt, the AR system may determine that the copying is authorized and, at block 1514 , display the copy of the content outside the volume 1522. If the user indicates that the copying is not authorized, at block 1512 , the AR system may determine not to display a copy of the content at the requested placement location.

[0161] The copy of the content may include a visual representation of the content 1526 that may or may not include metadata associated with the originating requestor and may or may not allow the requestor to control one or more parameters associated with the copy. For example, the copy may include a visual representation of the content 1526 that does not allow the party that originated the content 1526 (such as a network domain) to control movement, animation, or other aspects of the representation. In another example, the copy may include a representation of the content 1526 that includes metadata connected to the requestor, such as a hyperlink, animation, or other data associated with the requestor.

[0162] At block 1512, the AR system may hide or not display some or all of the content 1526, such as Figure 15B 1501. For example, content 1526 may be sized or oriented so that the center of content 1526 falls outside 3D volume 1522. If the center of content 1526 falls outside 3D volume 1522, the AR system may determine not to display the entire content 1526. In another example, the AR system may determine to display the portion of content 1526 that falls within 3D volume 1522.

[0163] K. Additional Examples

[0164] The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desired properties. Without limiting the scope of this disclosure, several non-limiting features will now be briefly discussed. The following paragraphs describe various example implementations of the devices, systems, and methods described herein.

[0165] Example 1: A display system for displaying virtual content in a three-dimensional (3D) spatial environment, the display system comprising: a head-mounted display configured to present the virtual content to eyes of a user of the display system; and circuitry in communication with the head-mounted display, the circuitry configured to: receive a request to access network-based 3D content; identify parameters associated with the network-based 3D content, comprising at least one of: a location in the user's 3D spatial environment at which the network-based 3D content is to be displayed, an orientation of the 3D content, or a size of the 3D content; based on the parameters, determine whether the 3D content can be displayed in an authorized portion of the user's 3D spatial environment; and in response to determining that the 3D content cannot be displayed in the authorized portion of the 3D spatial environment, adjust the size of the authorized portion to allow the 3D content to be displayed in the adjusted authorized portion.

[0166] Example 2: A display system according to Example 1, wherein the circuit is configured to: display the browser block in a 3D spatial environment, wherein the authorized portion of the 3D spatial environment includes: a width greater than the width of the browser block; a height greater than the height of the browser block; and a depth greater than the depth of the browser block.

[0167] Example 3: The display system of any of Examples 1 or 2, wherein the circuitry is configured to display the browser block in the authorization portion.

[0168] Example 4: The display system of any of Examples 1-3, wherein to determine whether the 3D content can be displayed in the authorized portion, the circuitry is configured to determine whether a threshold amount of the 3D content can be displayed within the authorized portion.

[0169] Example 5: The display system of any of Examples 1-4, wherein the circuitry is configured to: in response to determining that the 3D content can be displayed in the authorized portion of the 3D spatial environment, display the 3D content in the authorized portion.

[0170] Example 6: The display system of any of Examples 1-5, wherein to resize the authorized portion, the circuitry is configured to: determine a resize authorization status; and resize the authorized portion based on the resize authorization status.

[0171] Example 7: The display system of any of Examples 1-6, wherein to determine the resize authorization status, the circuitry is configured to identify the recognition authorization based on user input.

[0172] Example 8: The display system of any of Examples 1-7, wherein the network-based 3D content is associated with a network domain, and wherein, to determine the resizing authorization status, the circuitry is configured to identify authorization based on whether the network domain is an authorization requestor.

[0173] Example 9: The display system of any of Examples 1-8, wherein the resize authorization state comprises authorization to resize the authorized portion during a current session of accessing a network domain associated with the network-based 3D content.

[0174] Example 10: A display system according to any of Examples 1-9, wherein, to adjust the size of the authorization portion, the circuit is configured to: increase the width of the authorization portion by a first amount in at least one direction; increase the height of the authorization portion by a second amount in at least one direction; and increase the depth of the authorization portion by a third amount in at least one direction.

[0175] Example 11: The display system of any of Examples 1-10, wherein the circuit is configured to: determine a refresh condition associated with the authorized portion; and set the size of the authorized portion to a default size.

[0176] Example 12: The display system of Example 11, wherein the refresh condition comprises at least one of: user interaction with a network domain not associated with the network-based 3D content, and an instruction from the user to stop displaying the network-based 3D content.

[0177] Example 13: A display system for displaying virtual content in a three-dimensional (3D) spatial environment, the display system comprising:

[0178] A head-mounted display configured to present virtual content to an eye of a user of the display system; and circuitry in communication with the head-mounted display, the circuitry configured to: receive a request to access content; display the content in an authorized portion of the user's 3D spatial environment at a first orientation; receive a request to display the content in the user's 3D spatial environment at a second orientation; determine whether the content can be displayed in the authorized portion of the user's 3D spatial environment at the second orientation; and in response to determining that the content cannot be displayed in the authorized portion of the 3D spatial environment at the second orientation, resize the authorized portion to allow display of the content in the second orientation within the resized authorized portion.

[0179] Example 14: The display system of Example 13, wherein the content comprises a web page.

[0180] Example 15: A display system according to Example 13 or 14, the circuit being configured to: display content within a virtual browser block in a 3D spatial environment, and wherein the authorized portion of the 3D spatial environment includes: a width greater than the width of the virtual browser block; a height greater than the height of the virtual browser block; and a depth greater than the depth of the virtual browser block.

[0181] Example 16: The display system of Example 15, wherein the circuitry is configured to display the virtual browser tile in the authorization portion.

[0182] Example 17: The display system of any of Examples 13-16, wherein to determine whether the content can be displayed in the authorized portion, the circuitry is configured to determine whether a threshold amount of the content can be displayed within the authorized portion.

[0183] Example 18: The display system of any of Examples 13-17, wherein the circuitry is configured to: in response to determining that the content can be displayed in the authorized portion of the 3D spatial environment in the second orientation, display the content in the authorized portion in the second orientation.

[0184] Example 19: The display system of any of Examples 13-18, wherein to resize the authorized portion, the circuit is configured to: determine a resize authorization status; and resize the authorized portion based on the resize authorization status.

[0185] Example 20: The display system of any of Examples 13-19, wherein to determine the resize authorization status, the circuitry is configured to identify the authorization based on user input.

[0186] Example 21: The display system of any of Examples 13-20, wherein the content is associated with a network domain, and wherein, to determine the resize authorization status, the circuitry is configured to identify authorization based on whether the network domain is an authorized requestor.

[0187] Example 22: The display system of any of Examples 13-21, wherein the resize authorization state comprises authorization to resize the authorized portion during a current session of accessing the network domain associated with the 3D content.

[0188] Example 23: A display system according to any of Examples 13-22, wherein, to adjust the size of the authorization portion, the circuit is configured to: increase the width of the authorization portion by a first amount in at least one direction; increase the height of the authorization portion by a second amount in at least one direction; and increase the depth of the authorization portion by a third amount in at least one direction.

[0189] Example 24: The display system of any of Examples 13-23, wherein the circuit is configured to: determine a refresh condition associated with the authorized portion; and set the size of the authorized portion to a default size.

[0190] Example 25: The display system of Example 24, wherein the refresh condition comprises at least one of: user interaction with a network domain not associated with the content, and an instruction from the user to stop displaying the content.

[0191] Example 26: A display system for displaying virtual content in a three-dimensional (3D) spatial environment, the display system comprising: a head-mounted display configured to present the virtual content to eyes of a user of the display system; and circuitry in communication with the head-mounted display, the circuitry configured to: receive a request to access 3D content; identify parameters associated with the 3D content, comprising at least one of: a location in the user's 3D spatial environment at which to display the 3D content, an orientation of the 3D content, and a size of the 3D content; based on the parameters, determine whether the 3D content can be displayed in an authorized portion of the user's 3D spatial environment; and in response to determining that the 3D content cannot be displayed in the authorized portion of the 3D spatial environment, display a representation of the 3D content at the location in the 3D spatial environment.

[0192] Example 27: A display system according to Example 26, wherein the circuit is configured to: display the browser block in a 3D spatial environment, and wherein the authorized portion of the 3D spatial environment includes: a width greater than the width of the browser block; a height greater than the height of the browser block; and a depth greater than the depth of the browser block.

[0193] Example 28: The display system of any of Examples 26 or 27, wherein the circuitry is configured to display the browser block in the authorization portion.

[0194] Example 29: The display system of any of Examples 26-28, wherein to determine whether the 3D content can be displayed in the authorized portion, the circuitry is configured to determine whether a threshold amount of the 3D content can be displayed within the authorized portion.

[0195] Example 30: The display system of any of Examples 26-28, wherein the circuitry is configured to: in response to determining that the 3D content can be displayed in the authorized portion of the 3D spatial environment, display the 3D content in the authorized portion.

[0196] Any of the above examples may be combined in any suitable combination.

[0197] L. the term

[0198] To facilitate understanding of the systems and methods discussed herein, a number of terms are described below. The terms described below, as well as other terms used herein, should be interpreted to include the descriptions provided, the ordinary and customary meanings of the terms, and / or any other implied meanings used for the respective terms, where such interpretation is consistent with the context of the terms. Accordingly, the following descriptions do not limit the meanings of these terms, but rather provide example descriptions only.

[0199] Prism: A container, area, or volume associated with mixed reality content. For example, a prism can contain multiple virtual content items that can be selected by the user. A prism can be created when an app is launched, and can then spawn sibling or child prisms to create flexible layouts. Apps within a prism can be configured to control where these layered prisms will appear, typically near the first prism and easily discoverable by the user.

[0200] Prism can provide feedback to the user. In some embodiments, the feedback can be a title that is only displayed to the user when Prism is targeted with a head gesture. In some embodiments, the feedback can be a glow around the Prism. The Prism glow (and / or other Prism feedback) can also be used for sharing to give the user feedback about the Prism being shared.

[0201] Controller: A handheld controller, such as a totem.

[0202] Controller Axis: The axis extending from the controller that defines the direction the controller is pointing.

[0203] Head pose: Head position and / or head orientation determined using sensors such as an inertial measurement unit (IMU), accelerometer, gyroscope, etc. A head pose ray extending in the direction of the head pose can be used to interact with virtual objects. For example, when a user is pointing at or looking at a prism or object, the object or prism intersects the user's head pose ray.

[0204] Focus: A property of an object (such as a prism) that allows an interactive object to be selected.

[0205] Input Focus: A property of an object (such as a prism or application) that causes the object's cursor to be refreshed and rendered as the active system cursor. In some implementations, there can be multiple focus objects, but only one has input focus.

[0206] Browser Block: A content window that can be used to navigate, display, or otherwise interact with web-based content. Browser blocks can be displayed as 2D or 3D objects that can be manipulated and interacted with within the user's 3D environment.

[0207] 3D Content: Virtual objects that can be displayed within a user's 3D environment. 3D content can be static, animated, manipulable, or otherwise interactive. 3D content can include web-based content generated through user interaction with a web domain or web page using, for example, a browser.

[0208] Content volume (or content stage): A content volume may include a volume of space with a user's environment in which 3D content may be displayed or manipulated.

[0209] M. Other considerations

[0210] Each of the processes, methods, and algorithms described herein or depicted in the accompanying drawings can be embodied in or fully or partially automated by code modules executed by one or more physical computing systems, hardware computer processors, dedicated circuits, or electronic hardware configured to execute specific and concrete computer instructions. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, dedicated circuit, or the like programmed with specific computer instructions. The code modules can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some embodiments, specific operations and methods can be performed by circuits specific to a given function.

[0211] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that dedicated hardware or one or more physical computing devices (utilizing appropriate dedicated executable instructions) may be necessary to perform the functionality, for example, due to the volume or complexity of the calculations involved, or to provide results in substantially real time. For example, an animation or video may include many frames, each of which has millions of pixels, and may require specially programmed computer hardware to process the video data to provide the desired image processing task or application within a commercially reasonable amount of time.

[0212] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage devices (including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks), volatile or non-volatile storage devices, combinations thereof, etc. Methods and modules (or data) may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over various computer-readable transmission media (including wireless-based and wire / cable-based media) and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored permanently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0213] Any process, frame, state, step or function in the flowchart described herein or depicted in the accompanying drawings should be understood to potentially represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (such as logical or arithmetic) or step in the process. Various processes, frames, states, steps or functions can be combined, rearranged, added to the illustrative examples provided herein, deleted, modified or otherwise changed from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the frames, steps or states associated therewith can be executed in other appropriate orders (such as sequentially, in parallel or in some other manner). Tasks or events can be added to or deleted from the disclosed example embodiments. Moreover, the separation of the various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0214] The processes, methods, and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0215] The systems and methods of the present disclosure each have several innovative aspects, no single one of which is solely responsible for or requires the desired properties disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the embodiments described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments or examples shown herein, but will conform to the widest scope consistent with the disclosure, principles, and novel features disclosed herein.

[0216] Certain features described in this specification in the context of separate embodiments or examples may also be implemented in combination in a single embodiment or example. Conversely, various features described in the context of a single embodiment or example may also be implemented in multiple embodiments or examples, either individually or in any suitable subcombination. Moreover, although features may be described above as functioning in certain combinations, and even initially claimed as such, in some cases one or more features of the claimed combination may be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of subcombinations. No single feature or group of features is necessary or essential for every embodiment.

[0217] Conditional language used herein, particularly words such as "can," "would," "might," "may," "could," "for example," etc., unless expressly stated otherwise, is generally understood in the context in which it is used to convey that certain embodiments include and other embodiments do not include certain features, elements, and / or steps. Thus, such conditional language is generally not intended to imply that a feature, element, or step is in any way essential to one or more embodiments, or that one or more embodiments necessarily include logic for determining whether such features, elements, or steps are included or to be performed in any particular embodiment, with or without author input or prompting. The terms "comprise," "include," "have," etc. are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Furthermore, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, "a," "an," and "the" as used in this application and the appended claims should be interpreted to mean "one or more" or "at least one," unless otherwise stated.

[0218] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including individual members. For example, "at least one of A, B, or C" is intended to encompass: A, B, C, A and B, A and C, B and C, and A, B, and C. Unless specifically stated otherwise, words such as "at least one of X, Y, and Z" should be understood in conjunction with the context as generally used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such linking language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0219] Similarly, although operations may be depicted in the accompanying drawings in a particular order, it should be understood that such operations do not need to be performed in the particular order shown or in a sequential order, or that all illustrated operations need not be performed to achieve the desired result. Further, the accompanying drawings may schematically depict one or more example processes in the form of flow charts. However, other operations not depicted may be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the illustrated operations. In addition, in other implementations, operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, other implementations are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result.

Claims

1. A head-mounted display (HMD) system, comprising: Hardware Computer Processor; A non-transitory computer-readable medium having software instructions stored thereon, the software instructions being executable by the hardware computer processor to cause the computing system to perform operations including: receiving a request to display web-accessible content in a three-dimensional (3D) spatial environment depicted on a display of the HMD system, wherein the web-accessible content includes at least one two-dimensional (2D) web page; identifying parameters associated with the content, the parameters comprising at least one of: an initial position at which the content is displayed in the 3D spatial environment, an initial orientation of the content, or an initial size of the content; rendering the content within a bounded volume in the 3D spatial environment based on the identified parameters, including rendering the at least one 2D webpage on at least one first surface of the bounded volume; receiving a user selection of at least a portion of the content including the at least one 2D web page by a user of the HMD system; receiving a request initiated by the user to change the orientation of at least a portion of the selected content from the initial orientation to an updated orientation, the updated orientation comprising mapping to display the at least one 2D webpage on at least one second surface of the bounding volume, the at least one second surface being different from the at least one first surface; determining whether a threshold amount of the content can be displayed within the bounding volume of the 3D spatial environment at the updated orientation; responsive to determining that the threshold amount of the content cannot be displayed within the bounding volume of the 3D spatial environment at the updated orientation, determining whether resizing of the bounding volume is authorized by the user; responsive to determining that resizing of the bounding volume is authorized by the user, automatically resizing the bounding volume to display at least the threshold amount of the content in the updated orientation; Redirecting at least a portion of the selected content to the updated orientation based on the user request; and Rendering the at least a portion of the content at the updated orientation includes rendering the at least one 2D web page on the at least one second surface of the bounding volume.

2. The display system according to claim 1, wherein: The initial orientation is vertical, and the updated orientation is horizontal.

3. The display system according to claim 1, wherein: The initial orientation is horizontal, and the updated orientation is vertical.

4. The display system according to claim 1, wherein: The updated orientation is based on a posture of the user.

5. The display system according to claim 4, wherein: The gesture of the user includes one or more of the following: a head gesture, or an eye gesture.

6. The display system according to claim 4, wherein: The operations further include: tracking the gesture of the user; and In response to a change in the posture of the user, the at least a portion of the content is automatically re-orientated.

7. The display system according to claim 1, wherein: Determining authorization to resize the bounded volume includes prompting the user, via the display of the HMD system, to authorize the resizing.

8. The display system according to claim 1, wherein: Determining that resizing of the bounding volume is authorized includes determining that the user authorizes all resizing requests during a current interaction session with the HMD system.

9. The display system according to claim 1, wherein: Determining that resizing of the bounding volume is authorized includes determining that resizing is within a range of prior authorization by the user to resize the bounding volume.

10. The display system according to claim 1, wherein: The initial size of the bounding volume is a predetermined proportion of the user's field of view within the display.

Images