Body-Centered Content Positioning for 3D Containers in a Composite Reality Environment

Abstract

To provide body-centric content positioning relative to a three-dimensional container in a preferred mixed reality environment.SOLUTION: A mixed reality system is disclosed that displays a 3D content in a container referred to as a "prism," and also displays a related content item, such as a bidirectional menu for interacting with the 3D content. To maintain the content item accessible to a user, the system repositions the content item relative to the prism as the user moves relative to the prism. For example, in the context of an annotation toolbar, the system may move the toolbar from one surface of the prism to another in response to the user walking around the prism, entering the prism, or changing a head pose in the prism.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to augmented reality and mixed reality systems, and more particularly to user interfaces and processes for positioning content relative to a three-dimensional (3D) container, a so-called "prism," based on user movement.

Background Art

[0002] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality," "augmented reality," and "mixed reality" experiences, in which digitally reproduced images or portions thereof are presented to a user such that they appear as if they were real. Virtual reality, i.e., a "VR" scenario, typically involves the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality, i.e., an "AR" scenario, typically involves the presentation of digital or virtual image information as an augmentation to the visualization of the actual world surrounding the user. Mixed reality, i.e., an "MR" scenario, typically involves the fusion of the real and virtual worlds, producing a new environment in which physical and virtual objects coexist and interact in real time. As used herein, the term "mixed reality" is intended to encompass augmented reality.

[0003] As disclosed in U.S. Patent No. 2019 / 0197785 (which is incorporated by reference in its entirety), a mixed reality system may support the use of a 3D bounded volume or container, referred to as a "prism," to manage the display of virtual content. The prism may have a rectangular or other shape and may be positioned at a fixed location relative to the user's real-world environment. In some cases, multiple applications may render 3D content within the prism, and a universe application may manage the display of this content.

Summary of the Invention

Means for Solving the Problems

[0004] One problem faced in displaying multiple content items within a 3D environment involves potential interference between the content of different items, including the content displayed within the prism. As an example, some mixed reality applications expose a two-way menu, such as a toolbar, that can interact with the user to create or modify content displayed within the 3D environment. As the user moves around, the display of this menu may overlap or appear behind one or more content items, making interaction with the menu difficult. A related problem involves maintaining the two-way menu in a location where it can be easily seen and accessed as the user moves around.

[0005] The present disclosure addresses the above and other problems by providing a display process for adjusting the display position and / or orientation of content items relative to the boundaries of a prism (or other bounded 3D volume or container) as the user moves relative to the prism. The content items can be, for example, two-way display objects such as two-way menus (e.g., toolbars). As another example, the content items can be notifications, or panes that display display objects or notifications.

[0006] In some embodiments or use cases, the process involves repositioning content items from one surface of the prism to another surface of the prism based on the movement of the user relative to the prism and / or based on changes in the orientation of the user (e.g., head pose). For example, assume that the prism has a rectangular shape and is of a sufficient size for the user to walk in and out of the prism. If the user stands outside the prism initially and faces the prism, a two-way menu (or other content item) can be displayed on the closest vertical surface of the prism first, in an outward-facing orientation. If the user then walks into the prism through this surface, the two-way menu can be repositioned to the rear surface (i.e., the surface parallel to and directly opposite the surface through which the user walked and passed), and may be displayed in an inward-facing orientation. (As discussed below, display positions corresponding between these parallel surfaces are also potentially considered.) If the user then turns right or left so as to face one of the other two vertical surfaces of the rectangular prism, the two-way menu can be repositioned to the surface the user is facing. The process can thereby maintain the menu within the user's field of view while reducing or eliminating interference between the menu and other content rendered within the prism.

[0007] The process may also involve reserving one or more portions of the prism or its surface for the display of content items so that the application cannot render other types of content that would interfere with the content items. For example, with respect to a prism having multiple vertical surfaces, an upper portion or strip may be reserved for the display of the two-way menu. This feature can further reduce or eliminate the likelihood that content rendered within the prism may obstruct the user's ability to access the menu or other content items.

[0008] In some embodiments, the process advantageously facilitates the user's ability to access content items while the user moves relative to the prism and further keeps the content items "out of the way." For example, if the content item is an annotation toolbar for annotating 3D content rendered within the prism, the toolbar may (1) generally remain visible to the user (e.g., stay within the user's field of view) and (2) move between multiple set locations and orientations relative to the prism as the user moves around such that the 3D content and annotations within the prism do not obstruct the user's access to the toolbar.

[0009] The process may be implemented in executable program instructions executed by one or more hardware processors of a mixed reality system. For example, the process may be implemented within the executable code of a universe application running on the mixed reality system and may manage content rendered by other applications.

[0010] The present invention thus includes a computerized process that is executed by one or more hardware processors of a mixed reality system and is performed under the control of program instructions. The process includes displaying, on a display of a headset worn by a user, three-dimensional (3D) content contained within a prism, the prism having a plurality of surfaces that define a volume in which the 3D content is displayed; displaying, on a first surface of the prism, a content item that provides a function for the user to interact with the 3D content; sensing movement of the user with respect to the prism; and repositioning the content item to a second surface of the prism in response to the sensed movement. The content item may be, for example, an icon selectable by the user or a toolbar having an annotation menu that provides a function for the user to annotate the 3D content. The sensed movement may include a change in the user's location or a change in the user's head pose. The content item may be repositioned to remain within the user's field of view. In some embodiments, the prism includes a reserved region in which it is not possible to display the 3D content, and the process includes displaying the content item on the first and second surfaces within the reserved region.

[0011] In some embodiments of the computerized process, the first and second surfaces are perpendicular surfaces that are perpendicular to each other. In such embodiments, repositioning the content item may include rotating the content item by 90 degrees.

[0012] In some embodiments of the computerized process, the first and second surfaces are parallel to each other, and the sensed movement comprises a user movement from outside the prism to inside the prism through the first surface. In such embodiments, repositioning the content item comprises displaying the content item at an intermediate position corresponding between the first and second surfaces in response to the user movement from outside to inside the prism.

[0013] In some embodiments of the computerized process, the process comprises determining a display position of a content item based on a user location when the user is located outside the prism, and determining a display position of the content item based on a user's pose when the user is located inside the prism.

[0014] The present invention also provides a computing system comprising a head-mounted display system configured to be worn by a user, one or more sensors configured to sense movement of the user, and one or more processors. The computing system is programmed with executable instructions for displaying 3D content contained within a prism on the head-mounted display system. The prism has a surface that defines a bounded volume and has a fixed location within the user's real-world environment. The computing system comprises a composite reality system including a computer system. The computing system is configured to maintain content items accessible to the user by displaying a content item on a first surface of the prism and repositioning the content item to a second surface of the prism in response to a sensed movement of the user relative to the prism. The content item may be, for example, a two-way menu. The sensed movement may include a change in the user's location relative to the prism, a movement of the user from outside the prism to inside the prism, and / or a change in the user's posture. The prism may include reserved areas where 3D content cannot be displayed, in which case the computing system may be programmed to display content items on the first and second surfaces within the reserved areas. In some embodiments, the computing system is programmed to determine a display position of a content item based on the user's location when the user is located outside the prism and based on the user's posture when the user is located inside the prism. The computing system may execute a first application that renders 3D content and a second application that manages the display of the 3D content within the prism.

[0015] The present invention further provides a non-transitory computer storage device comprising one or more memory device(s), the non-transitory computer storage device storing executable program instructions for instructing to perform a process including: displaying, on a display of a headset worn by a user, 3D content contained within a prism to the user, the prism having a plurality of surfaces defining a volume in which the 3D content is displayed; displaying, on a first surface of the prism, a content item providing a function for the user to interact with the 3D content; detecting movement of the user with respect to the prism; and in response to the detected movement, repositioning the content item to a second surface of the prism so that the content item remains within the user's field of view. The present invention provides, for example, the following. (Item 1) A computerized process performed under the control of program instructions executed by one or more hardware processors of a mixed reality system, the process comprising: displaying, on a display of a headset worn by a user, 3D content contained within a prism to the user, the prism having a plurality of surfaces defining a volume in which the 3D content is displayed; displaying, on a first surface of the prism, a content item providing a function for the user to interact with the 3D content; detecting movement of the user with respect to the prism; and in response to the detected movement, repositioning the content item to a second surface of the prism. A process as described above. (Item 2) The process according to item 1, wherein the detected movement comprises a change in the location of the user. (Item 3) The perceived movement comprises a change in the user's head pose, and the content item is repositioned to remain within the user's field of view, the process of claim 1. (Item 4) The prism comprises a reserved area in which the 3D content cannot be displayed, and the process includes displaying the content item on the first and second surfaces within the reserved area, the process of claim 1. (Item 5) The first and second surfaces are perpendicular surfaces that are perpendicular to each other, the process of claim 1. (Item 6) Repositioning the content item includes rotating the content item by 90 degrees, the process of claim 5. (Item 7) The first and second surfaces are parallel to each other, and the perceived movement comprises the user's movement from outside the prism to inside the prism through the first surface, the process of claim 1. (Item 8) Repositioning the content item includes displaying the content item at an intermediate position corresponding to between the first surface and the second surface in response to the user's movement from outside the prism to inside the prism, the process of claim 7. (Item 9) The content item is a toolbar having icons selectable by the user, the process of claim 1. (Item 10) The content item is an annotation menu that provides the user with a function for annotating the 3D content, the process of claim 1. (Item 11) The process according to item 1, wherein the process includes determining a display position of the content item based on a user location when the user is located outside the prism, and determining the display position of the content item based on the user's posture when the user is located inside the prism. (Item 12) A mixed reality system, A head-mounted display system configured to be worn by a user, One or more sensors configured to sense movement of the user, A computing system comprising one or more processors, the computing system being programmed with executable instructions for displaying three-dimensional (3D) content contained within a prism on the head-mounted display system, the prism having a surface defining a bounded volume, the prism having a fixed location within the user's real-world environment. Comprising The computing system is configured to maintain the content item accessible to the user by displaying the content item on a first surface of the prism and repositioning the content item to a second surface of the prism in response to a sensed movement of the user relative to the prism. (Item 13) The mixed reality system according to item 12, wherein the content item is a two-way menu. (Item 14) The mixed reality system according to item 12, wherein the sensed movement comprises a change in the user's location relative to the prism. (Item 15) The mixed reality system according to item 12, wherein the sensed movement comprises the user's movement from outside the prism to inside the prism. (Item 16) The composite reality system of item 12, wherein the perceived movement comprises a change in the user's posture. (Item 17) The composite reality system of item 12, wherein the prism has a reserved area in which the 3D content cannot be displayed, and the computing system is programmed to display the content items on the first and second surfaces within the reserved area. (Item 18) The composite reality system of item 12, wherein the computing system is programmed to determine the display position of the content items based on the user's location when the user is located outside the prism, and to determine the display position of the content items based on the user's posture when the user is located inside the prism. (Item 19) The composite reality system of item 12, wherein the computing system executes a first application for rendering the 3D content and a second application for managing the display of the 3D content within the prism. (Item 20) A non-transitory computer storage device comprising one or more storage device devices, the non-transitory computer storage device storing executable program instructions, the executable program instructions causing a composite reality system to display to the user, on a display of a headset worn by the user, three-dimensional (3D) content contained within a prism, the prism having a plurality of surfaces defining a volume in which the 3D content is displayed, display, on a first surface of the prism, a content item that provides a function for the user to interact with the 3D content, detect movement of the user with respect to the prism, Responsive to the detected movement, repositioning the content item on the second surface of the prism so that the content item remains within the user's field of view A non-transitory computer storage device that instructs to perform a process including

Brief Description of the Drawings

[0016]

Figure 1

[0017]

Figure 2

[0018]

Figure 3

[0019]

Figure 4

[0020]

Figure 5

[0021]

Figure 6

[0022]

Figure 7

[0023]

Figure 8

[0024] Throughout the drawings, reference numerals may be reused to indicate corresponding between the elements being referenced. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION

[0025] Detailed Description I. Terminology To facilitate understanding of the systems and methods discussed herein, several terms are described below. These terms, as well as other terms used herein, are to be interpreted as including the provided description, the terms in their ordinary and customary meaning, and / or any other implied meaning regarding the individual terms, such that such construction is consistent with the context of the terms. Accordingly, the following description does not limit the meaning of these terms but provides only an illustrative description.

[0026] Head pose: The position and / or orientation of a user's head in the real world, represented by a wearable headset (or other head-mounted system such as a helmet), or its proxy. The head pose may be determined using sensors such as an inertial measurement unit (IMU), accelerometer, gyroscope, etc. In some embodiments, a head pose ray extending in the direction of the head pose may be used to interact with virtual objects. For example, when the user is pointing at or looking at a prism or object, the object or prism is intersected by the user's head pose ray. The user's orientation may additionally or alternatively be determined based on eye pose, body pose, or the pose of some other part of the user's body.

[0027] Prism: A three-dimensional container or enclosure associated with mixed reality content or a mixed reality space. For example, a prism may contain one or more virtual content items that may be selectable by the user. The boundaries or surfaces of the prism may or may not be displayed to the user. A prism may spawn when an application is launched and then spawn sibling or child prisms to create a flexible layout. A prism typically serves a function similar to that of a window of a graphical user interface but is a 3D object that can be displayed within 3D content. Prisms can be used in some environments to enable users to collaboratively share and manipulate 3D content.

[0028] Controller: A device configured to be held by hand and used to interact with a mixed reality environment. For example, the controller may be used to interact with content items rendered using prisms or menus associated with prisms. The controller may provide multiple degrees of freedom of movement, such as 6DoF (6 degrees of freedom). In some embodiments, the user's hand or other real-world object may be used instead of or in addition to the controller to provide user input to, for example, virtual content that the user can interact with.

[0029] Controller pose: The position and / or orientation of the controller. The controller pose can be used to determine the area, volume, or point in the mixed reality environment at which the controller points. In some embodiments, the controller pose is visually depicted to the user as a ray extending from the controller, enabling the user to point at and select content items using the controller. II. Exemplary Mixed Reality Environments with Prisms

[0030] FIG. 1 shows an exemplary physical environment and computing system for managing and displaying content using a prism in a mixed reality system. The representative environment 100 includes the user's landscape 110 as viewed by the user 103 through a head-mounted system or headset 160. The user's landscape 110 is a 3D view of the world onto which user-installed content can be synthesized over the real world. The headset 160 can be any suitable head-mounted display system or device, for example, similar goggles, glasses, or a helmet worn by the user.

[0031] The headset 160 includes a display 162 that displays various types of content that can complement the user's view of the real world. In the embodiment shown in FIG. 1, a user standing in a room and viewing a landscape 110 that includes a real-world view of the room is complemented by digital content 115a and 115b rendered within individual prisms 113a, 113b. The display 162 can be integrated within a transparent or translucent lens (or one such lens) of the headset, in which case the user's view through the lens is complemented by the content rendered on the display 162. For example, the display 162 may be implemented using a waveguide incorporated within the lens, as disclosed in U.S. Patent Publication No. 2018 / 0157398, the disclosure of which is incorporated herein by reference. Alternatively, the display 162 may be an opaque display that displays a complemented representation of the real world view, such as that captured by one or more outward-facing cameras of the headset.

[0032] The headset 160 includes one or more types of sensors 164 for sensing the user's landscape, position, and orientation. The sensors 164 can include, for example, one or more outward-facing cameras or scanners for sensing physical objects and other surroundings, and one or more types of devices such as accelerometers, inertial measurement units, compasses, wireless devices, and / or gyroscopes for sensing head pose and location. The headset may also include one or more inward-facing cameras or other sensors for eye tracking. The tracked state of the user's eyes is referred to as the "eye pose." The headset 160 may also include audio speakers and microphones (not shown).

[0033] The representative environment 100 further includes a wearable computing device 170. The wearable computing device may include one or more hardware processors (not shown), a memory or other storage device 137 for storing application code and content, and one or more wireless transceivers for communicating with local and / or remote system components. Some or all of the circuitry of the wearable computing device may alternatively be integrated within the headset 160 or located in a stationary computing device that is not worn by the user. The wearable computing device 170 may communicate with a remote server or a local processing node via a wireless network. In some embodiments, the wearable computing device 170 is connected to the headset 160 by a cable and is configured to process video and sensor data received from the headset 160. In FIG. 1, as depicted by the external database (DB) 150, the wearable computing device 170 may access an external storage device via a network.

[0034] The term "computing system" is used herein to collectively refer to the various hardware computing components (hardware processors, storage devices, etc.) of a system, regardless of whether these components are contained within a single computing device or node or are distributed across multiple computing devices or nodes. In some embodiments, all of the processing components of the system are configured to be worn by a user such that the computing system is a wearable computing system.

[0035] In the embodiment of FIG. 1, the wearable computing device 170 executes two types of applications, namely, the universe application 130 and one or more additional applications 140 (sometimes referred to as "landscape applications"). The application 140 generates content, including the content 115a, 115b rendered within the prisms 113a, 113b. In the illustrated embodiment, each prism 113a, 113b is a bounded rectangular volume and is pinned or anchored to a real-world object or structure. Prism 113a is anchored to wall 117a and displays virtual content 115a that appears to the user as being attached to the wall. Prism 113b is anchored to table 117b and displays virtual content 115b in the form of a 3D tree that appears to the user as being located on the table. As shown in additional figures and discussed below, some prisms may be sized and positioned to enable the user to walk into and out of the prism.

[0036] The universe application 130 is involved in creating the prisms 113a, 113b and managing the display of the digital content 115a, 115b. The universe application 130 may create a prism, for example, when the application 140 needs to render virtual content on the user's landscape 110. Multiple applications 140 may render content within the same prisms 113a, 113b. Each prism may have characteristics and properties that enable the universe application 130 to manage the placement and display of content in the composite reality environment by managing the prism itself. Two different types of applications are shown in FIG. 1, but the present invention does not require two different types of applications. Additionally, the boundary lines of the prisms are shown in FIG. 1 and subsequent drawings, but the boundary lines may or may not be displayed to the user.

[0037] The mixed reality system may also include a controller 206 (see FIG. 2) that is held by the user and can be used to perform various functions. The controller 206 may include one or more input devices or elements, such as buttons, touch pads, rotatable wheels, etc. In one embodiment, the controller also serves as a virtual laser pointer that can be used by the user to select an object. For example, the user can use the controller 206 to select menu options on a menu (such as the toolbar 208 shown in FIG. 2) by pointing the controller at the desired option. A virtual laser or light beam (not shown) extending from the front of the controller may be displayed to the user to facilitate the selection action. In some embodiments, the user can alternatively use a finger or a head pose light beam to select menu options.

[0038] In some scenarios where the user interacts with the content displayed within the prism, it is desirable to expose the user to content items that remain visible and accessible as the user walks around or within the prism. For example, when the user creates, edits, or annotates content within the prism, it may be desirable to display a menu (such as a toolbar) or other two-way content items in the vicinity of the prism and dynamically reposition this two-way content item so that it remains accessible as the user moves around. As another example, in the context of a collaborative session in which the user collaborates with other users to generate or modify content within the prism, it may be desirable to display a notification pane that displays information about other participants and dynamically reposition it.

[0039] The present disclosure addresses this problem by providing a process for dynamically repositioning content items (such as, but not limited to, a menu or a notification pane) in response to changes in a user's location and / or orientation. In some embodiments, the process involves repositioning a content item from one surface of a prism to another surface of the prism based on the user's movement relative to the prism and / or based on a change in the user's orientation (e.g., head pose). The content item may also be positioned in other locations, such as locations within the prism, in some scenarios. The process may also reorient (e.g., rotate horizontally by 180 degrees) the content item to maintain its proper orientation with respect to the user.

[0040] In one embodiment, the process is implemented within the executable instructions of the universe application 130. The process may alternatively be implemented in one of the executable codes of the application 140, the operating system, or another type of software component. III. Exemplary Repositioning of a Toolbar-Type Menu

[0041] Exemplary use case scenarios will be described here with reference to FIGS. 2-5. In this use case scenario, the user interacts with (e.g., creates, edits, or annotates) a 3D model 202 of an aircraft. The model 202 is positioned inside a prism 200, and the position and orientation of the model remain fixed with respect to the real / physical world as the user moves around. The content item in this embodiment is an annotation menu, specifically, an annotation toolbar 208 that includes a plurality of selectable icons (shown as circles), each of which can be selected by the user by pointing the controller 206 at the icon. Different icons can represent different annotation tools such as drawing tools, highlighting tools, erasing tools, text entry tools, etc. In other scenarios, the annotation menu may be in the form of a text menu that has a plurality of drop-down sub-menus. The annotation toolbar 208 is a 2D object and appears flat to the user, but in some embodiments, it means that the object can alternatively be 3D. In FIGS. 2-5, the lines indicating the boundaries or edges of the prism are typically not displayed to the user, but in some embodiments, they may be displayed. In FIG. 2, the toolbar 208 can be described as being "displayed on" the surface of the prism, generally meaning its position and orientation coincide with that of the surface of the prism. The phrase "displayed on the surface" is not intended to imply that the surface itself is visible to the user. In some embodiments, "displayed on the surface" may mean that the plane created by the content item can be aligned with the plane created from the face of the prism.

[0042] In the embodiment of FIG. 2-5, the toolbar 208 is always displayed within the upper region 210 of the prism, but is also displayed at different locations within this region depending on the location and orientation of the user. In some embodiments, this upper region 210 is reserved for the display of menus (such as toolbars) and / or other types of content items that are dynamically repositioned as the user moves around the surroundings. Thus, other types of content such as the model 202, any annotations added by the user, and other types of user-created content cannot be located within the upper region 210 in the illustrated embodiment.

[0043] This use of the reserved region 210 reduces or eliminates interference between user-created content and the toolbar 208. For example, user-created content does not obstruct the user's view or the user's ability to select toolbar icons of the toolbar 208. Although the reserved region 210 is used in the embodiment of FIG. 2-5, the disclosed process can be implemented without using a reserved region. Additionally, different portions of the prism, such as the lower portion or side surface of a vertical surface, can also be used as the reserved region.

[0044] In FIG. 2, user 103 stands outside and faces the prism 200. The annotation toolbar 208 is displayed in an outward-facing orientation at the center position on the surface of the prism closest to the user (within the reserved area 210). This enables the user to easily browse and access the toolbar 208 while viewing the model 202. The user may use the controller 206, for example, to select a drawing tool from the annotation toolbar 208 and then use the controller to draw on or around the model (as indicated by the two wavy lines 300 in FIG. 3). In the illustrated embodiment, the toolbar 208 cannot move along the surface on which it is shown in FIGS. 2 and 3 as the user's head posture changes. In some embodiments, and in the alternative embodiments described below, the toolbar can "slide" horizontally along the surface on which it is shown.

[0045] Figure 4 illustrates how toolbar 208 can be repositioned as user 103 walks towards and then into prism 200. Initially, the toolbar is positioned at location 208a, which is the position shown in FIGS. 2 and 3. As the user approaches prism 200, toolbar 208 is repositioned to an intermediate position that corresponds to the middle between the front and back of prism 200 from the user's line of sight. This enables the toolbar 208 to remain visible and accessible to the user. The transition from location 208a to 208b, and transitions to other locations described herein, can occur immediately or more gradually (e.g., by using an animation to show the toolbar moving from one location to another). After user 103 steps into prism 200 and continues walking forward, annotation menu 208 is repositioned to the back of the prism as shown by position 208c. The orientation of the toolbar remains unchanged, and the toolbar thus has an orientation facing inwards at position 208c. After the user continues walking forward through the back, the toolbar may be rotated 180 degrees horizontally so as to have an outward-facing orientation.

[0046] Only a single intermediate position 208b is shown in FIG. 4, but multiple intermediate positions may be used. The number of intermediate positions used may be based on the dimensions of the prism, if applicable. Additionally, the process may be implemented without the use of intermediate display positions; for example, toolbar 208 may always be displayed on one of the surfaces of the prism.

[0047] Figure 5 illustrates five different user positions and orientations and shows five corresponding toolbar locations and orientations. The first three embodiments shown in Figure 5 are the same as those shown in Figure 4. Embodiment 4 shows user 103 in prism 200 facing the side of the prism and shows a toolbar displayed on the side with an inward-facing orientation. The toolbar may be repositioned to its original location, for example, if the user turns 90 degrees to the right after walking into the prism. The orientation of the user is determined by the head pose, but alternatively may be determined based on eye pose, body, or torso pose, or some other orientation measurement. Embodiment 5 in Figure 5 shows user 103 standing outside the prism 200 facing the other side and shows toolbar 208 positioned on the side with an outward-facing orientation. In the illustrated embodiments of Figure 5, toolbar 208 is displayed at a fixed central position while it is displayed on any given surface or intermediate position of the prism, but in some embodiments, other positions may be used, such as aligned to the right, aligned to the left, offset from a corner or side such as area 210, etc.

[0048] As illustrated by these embodiments, in addition to maintaining the visibility and accessibility of toolbar 208, the process requires maintaining the toolbar positioned at a convenient distance from the user. For example, when the user is at user position 5 in Figure 5, the toolbar may alternatively be displayed at toolbar position 4, but a significant distance between the user and the toolbar is likely to impede the user's ability to view and select from the toolbar, especially if the prism is large. In some embodiments, the process may require maintaining the separation between toolbar 208 and headset 160 within a selected range such as 2 - 12 feet, 3 - 10 feet, or 4 - 8 feet. The toolbar may be removed from the user's landscape if the user is located further than a threshold distance from prism 200.

[0049] The display process illustrated in FIG. 2-5 may be adapted for use in combination with other prism shapes and configurations. For example, if the prism is in the shape of a pentagon such that it is viewable from above, the process may select from one of the five vertical surfaces based on the user's position and orientation. If the prism includes a curved surface, the display process may warp the toolbar or other 2D content item to conform to the curved prism surface on which it is displayed.

[0050] FIG. 6 illustrates the coordinate system and associated references that may be used by the system to programmatically select the toolbar / menu display position in the embodiment of FIGS. 2-5. The drawing includes a top view of the prism 200, showing the X and Z axes of the coordinate system with respect to the prism. The Y axis is the vertical axis and is thus not shown. The center of the prism is the point at X = 0 and Z = 0. The length of the prism in the X direction is represented as L X and the length of the prism in the Z direction is represented as L z Two area dividing lines or boundary lines 600, 602 are shown, each intersecting two opposite vertices of a rectangle. These dividing lines are used to define four areas shown as areas 1, 2, 3, and 4. Each of these areas is further subdivided into an inner (i.e., inside the prism) and an outer area. In this drawing, the user 103 is shown outside area 4 and the toolbar (menu) 208 is positioned on the prism surface of area 4 in an outward facing orientation. The toolbar may remain fixed in place as long as the user stays within a threshold distance from the line dividing area 4 into its inner and outer areas, or outside area 4.

[0051] In one embodiment, the toolbar 208 is only displayed on the vertical surface of the prism (the intermediate display position is not used) and is displayed according to the following rules. (1) When the user is outside the prism, the toolbar is displayed on the surface in an outward-facing orientation corresponding to the area where the user is located. (2) When the user is inside the prism, the toolbar is displayed on the surface the user is facing (e.g., the surface intersected by the direction vector or light ray representing the user's head pose) in an inward-facing orientation. These rules may be appropriately modified or extended to support the display at one or more intermediate display positions such as position 2 in FIG. 5.

[0052] Figures 7A - 7D show four internal user locations and orientations and four corresponding menu / toolbar locations. The direction vector 212 in these figures represents a user direction or pose vector such as a head pose vector. In any case, the menu 208 is displayed on the vertical prism surface intersected by the direction vector 212 in an inward-facing orientation regardless of the four internal regions where the user is located. In FIG. 7A, the user is facing the surface of region 2 and the menu 208 is displayed on the surface at a menu angle of zero. In FIG. 7B, the user is facing the surface of region 1 and the menu 208 is displayed on the surface at a menu angle of 3π / 2 radians or 270 degrees. In FIG. 7C, the user is facing the surface of region 3 and the menu 208 is displayed on the surface at an angle of π / 2 radians or 90 degrees. In FIG. 7D, the user is facing the surface of region 4 and the menu 208 is displayed on the surface at an angle of π radians or 180 degrees. In other words, in some embodiments, the menu 208 may rotate towards the user regardless of the location where the user is located. As explained above, one or more intermediate display locations may also be used when the user is inside or near the prism. For example, in FIG. 7B, if the distance between the user and the surface of region 1 exceeds a threshold, the menu 208 may alternatively be displayed at an intermediate position corresponding to the distance between the user and that surface.

[0053] As described above, when a change in the user's orientation and / or location triggers a change in the location and / or orientation of the toolbar 208, an animation can optionally be used to show the toolbar moving and / or rotating to its new location and / or orientation. To further improve the user experience, a hysteresis type function may be used to prevent the toolbar 208 from "jumping" back and forth between locations in a short time interval in response to minor user movement. For example, in FIG. 6, if the user is standing along the boundary line 600 between the outside of regions 1 and 4, the hysteresis function may prevent the menu 208 from jumping between the surfaces of regions 1 and 4 in response to minor user movement. The thickness of each of the boundary lines 600 and 602 in FIG. 6 represents the degree of hysteresis used. When the user is inside the prism, the hysteresis function may prevent the menu 208 from rapidly moving back and forth between surfaces as the user turns towards the corners of the prism. Hysteresis may additionally or alternatively be implemented by imposing a time-based limit that prevents the menu 208 from being repositioned again for a certain period (e.g., 1 or 2 seconds) once its location has been changed.

[0054] Figure 8 illustrates a process that can be used to relocate and reorient toolbar, another type of menu, or another type of content item based on user movement. The process may be implemented in program code executed by a computing system (e.g., the system's wearable computing device 170 and / or another processing component). For example, the process may be implemented in program code of the universe application 130 (FIG. 1) or another application 140. Although the process is illustrated as being executed every frame (e.g., 60 times per second if a frame rate of 60 frames per second is used), it may not be executed as frequently (e.g., every other frame or every third frame). In decision block 802 of FIG. 8, the process determines whether the user is inside the prism 200 based on the user's coordinates and the coordinates and dimensions of the prism 200. The user's coordinates may be based on data collection or generated by the headset 160 (or in some embodiments, by the controller 206 or another component).

[0055] If the user is inside the prism, the area in which content items can be displayed is determined based on a direction vector (block 804) that represents the direction the user is facing, as described above with reference to FIGS. 7A-7D. The direction vector may be based on head pose, but alternatively may be based on eye pose, body pose, or some other pose, or orientation measurement. In block 805, the process uses the determined area to calculate a rotation for displaying content items on the surface of the area in an inward-facing orientation.

[0056] If the user is outside the prism, the process determines an area based on the user's location (block 810), as discussed with reference to FIG. 6. In block 812, the process then calculates a rotation for displaying content items on the surface of the identified area in an outward-facing (i.e., facing the user) orientation.

[0057] In block 814, the process calculates the X and Z coordinates based on the region. For example, the process may calculate these coordinates so that the content item can be centered on the identified surface. The Y coordinate may be a fixed value corresponding to the reserved region 210 as described above. In blocks 816 and 818, the process sets a new rotation if it has changed and sets a new position (X and Z coordinates) if it has changed. As described above, an animation may be executed to display the content item moving to its new position and orientation.

[0058] The following equations can be used to determine the regions in blocks 804 and 810 of FIG. 8, where X and Z are coordinates, and L Z is the prism length in the Z direction (see FIG. 6), and L X is the prism length in the X direction, and δ is an optional hysteresis value. A = X(L Z ) B = Z(L X ) Region 1 = (A - B)>δ and (A + B)>δ Region 2 = (A - B)>δ and (A + B)< -δ Region 3 = (A - B)< -δ and (A + B)< -δ Region 4 = (A - B)< -δ and (A + B)>δ Region 0 = where it does not fall into any of Regions 1 - 4

[0059] The hysteresis value δ may be set to a value, for example, in the range of 0 to 0.8, using a zero value that provides no hysteresis and a larger value that provides more hysteresis. Region 0 is the region represented by the region division boundary lines 600 and 602 shown in FIG. 6, and the thickness or width of these bars corresponds to δ. To implement the hysteresis function in FIG. 8, the region determined by block 804 or 810 is region 0, and no action is taken (i.e., the rotation and position are not updated at blocks 816 and 818). Table 1 shows the angles used to display content items for each of the five regions when the user is outside the prism, and Table 2 shows the angles used when the user is inside the prism. [Table 1] [Table 2]

[0060] The process of FIG. 8 may be appropriately modified to enable content items to be displayed at an intermediate position between the surfaces as described above.

[0061] As will be apparent, the display process described above can be varied in a number of ways. For example, in one embodiment, when the user is at the prism 200, the toolbar 208 "slides" left and right (horizontally) along the prism surface the user is facing as the user moves, such that the toolbar stays in front of the user within the reserved region 210 (e.g., based on the head pose). When the toolbar reaches a corner of the prism, it may jump to an adjacent surface or be folded 90 degrees so that the slide can continue while a portion of the toolbar is displayed on one surface and the remaining portion is displayed on an adjacent surface. The left and right movement may also be implemented when the user is outside the prism.

[0062] In another embodiment, when the user is inside the prism 200, the toolbar 208 or other content items can be continuously displayed at centered locations on all four vertical surfaces or on two opposing surfaces, and thus, in this embodiment, the toolbar 208 does not reposition when the user is in the prism, and the user can view and use any one of a plurality of instances of the toolbar. As another example, the size of the toolbar can be decreased as the user moves towards the surface on which it is displayed, and can be increased as the user moves away from the surface. IV. Conclusion

[0063] All of the processes and tasks described herein may be implemented and fully automated by a computing system that includes one or more computing devices (e.g., wearable computing device 170, headset 160, and / or other devices). Each such computing device typically includes a hardware processor (or multiple hardware processors) that executes program instructions, or modules stored in memory or other non-transitory computer-readable storage media. The various functions disclosed herein may be embodied in such program instructions, but some or all of the disclosed functions may alternatively be implemented in application-specific circuitry of the computing system (e.g., ASIC or FPGA). If the computing system includes multiple computing devices, these devices need not be co-located.

[0064] The systems and methods of the present disclosure each have several innovative aspects, none of which alone contribute to or are required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0065] In particular, conditional clauses used herein such as "can", "could", "might", "may", "e.g.", and equivalents, generally convey that while one embodiment includes certain features, elements, and / or steps, other embodiments do not include them, unless specifically stated otherwise or understood otherwise within the context in which they are used. Thus, such conditional clauses are not generally intended to imply that the features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, regardless of the author's input or prompting. The terms "comprising", "including", "having", and equivalents are synonyms and are used inclusively in a non-limiting manner, without excluding additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. Additionally, the articles "a", "an", and "the" as used in this application and the appended claims should be construed to mean "one or more" or "at least one" unless otherwise defined.

[0066] As used herein, the phrase referring to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in a context such that, unless specifically stated otherwise, they are used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such connective phrases are generally not intended to imply that an embodiment requires the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0067] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not necessary for achieving the desired results, and that such operations may be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations need not be performed. Further, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described are generally integrated together in a single software product or may be packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results.

Claims

**Claim 1** A mixed reality system comprising a head - mountable display configured to be worn by a user, one or more sensors configured to sense movement of the user, a computing system comprising one or more processors, the computing system being programmed with executable instructions to display three - dimensional (3D) content contained within a prism on the head - mountable display, the prism having a surface defining a bounded volume, the prism having a fixed position within the user's real - world environment, and wherein the computing system is configured to display an item associated with the 3D content at a first fixed position relative to the prism and to maintain the item accessible to the user by re - positioning the item to a second fixed position relative to the prism in response to a sensed movement of the user relative to the prism, the second fixed position being at least partially based on the user's pose vector while the user is inside the prism, the item being different from the 3D content, wherein the first fixed position corresponds to an intermediate position between a first surface and a second surface of the prism, a mixed reality system. **Claim 2** The mixed reality system of claim 1, wherein the second fixed position is on the second surface of the prism and the sensed movement comprises the user entering the prism through the first surface. **Claim 3** The mixed reality system of claim 1, wherein the sensed movement comprises movement of the user from outside the prism to inside the prism. **Claim 4** The mixed reality system of claim 1, wherein the sensed movement comprises a change in the user's pose while the user is inside the prism. **Claim 5** The mixed reality system of claim 1, wherein the item is a two - way menu providing a function for interacting with the 3D content. **Claim 6** The composite reality system according to claim 1, wherein the first fixed position and the second fixed position are within a reserved area of the prism reserved for content other than the 3D content.

7. A computerized process implemented under the control of program instructions executed by one or more hardware processors of a composite reality system, the process comprising: displaying, on a display of a headset worn by a user, three-dimensional (3D) content contained within a prism to the user, the prism having a plurality of surfaces defining a volume within which the 3D content is displayed; displaying an item associated with the three-dimensional (3D) content at a first fixed position relative to the prism, the item being different from the 3D content; sensing movement of the user relative to the prism; responsive to the sensed movement, repositioning the item to a second fixed position relative to the prism, the second fixed position being at least partially based on the user's posture while the user is inside the prism; comprising A computerized process, wherein the first fixed position and the second fixed position are within a reserved area of the prism reserved for content other than the 3D content.

8. The computerized process according to claim 7, wherein the first fixed position is on a surface of the prism.

9. The computerized process according to claim 7, wherein the sensed movement comprises movement of the user from outside the prism to inside the prism.

10. The computerized process according to claim 7, wherein the sensed movement comprises a change in the user's posture while the user is inside the prism.

11. The computerized process according to claim 7, wherein the item is a two-way menu providing a function for interacting with the 3D content.

12. A composite reality system, comprising: a head-mountable display configured to be worn by a user; One or more sensors configured to sense movement of the user A computing system comprising one or more processors, the computing system being programmed with executable instructions for displaying three-dimensional (3D) content contained within a prism on the head-mountable display, the prism having a surface defining a bounded volume, the prism having a fixed position within the user's real-world environment, the computing system comprising The computing system is configured to display an item associated with the three-dimensional (3D) content at a first fixed position relative to the prism and to maintain the item accessible to the user by repositioning the item to a second fixed position relative to the prism in response to a sensed movement of the user relative to the prism, the first fixed position and the second fixed position corresponding within a reserved area of the prism, the reserved area being reserved for content other than the 3D content, a mixed reality system

13. The mixed reality system according to claim 12, wherein the first fixed position is on the surface of the prism

14. The mixed reality system according to claim 12, wherein at least one of the fixed positions corresponds to an intermediate position between two surfaces of the prism

15. The mixed reality system according to claim 12, wherein the reserved area is the uppermost portion of the bounded volume

16. A mixed reality system A head-mountable display configured to be worn by a user One or more sensors configured to sense movement of the user A computing system comprising one or more processors, the computing system being programmed with executable instructions for displaying three-dimensional (3D) content contained within a prism on the head-mountable display, the prism having a surface defining a bounded volume, the prism having a fixed position within the user's real-world environment, the computing system comprising The computing system is configured to display an item associated with the three-dimensional (3D) content at a first fixed position relative to the prism and to maintain the item accessible to the user by repositioning the item to a second fixed position relative to the prism in response to a perceived movement of the user relative to the prism, the second fixed position being at least partially based on a pose vector of the user while the user is inside the prism, the item being different from the 3D content, the perceived movement comprising a change in the pose of the user while the user is inside the prism, the computing system responding to the change in pose by repositioning the item from a first surface of the prism to a second surface of the prism, the second surface being perpendicular to the first surface, augmented reality system. **Claim 17** The augmented reality system according to claim 16, wherein the item is a two-way menu providing a function for interacting with the 3D content. **Claim 18** The augmented reality system according to claim 16, wherein the first fixed position and the second fixed position are within a reserved area of the prism reserved for content other than the 3D content. **Claim 19** The augmented reality system according to claim 18, wherein the reserved area is the uppermost portion of the bounded volume.

Images