Method and system for managing and displaying virtual content in a mixed reality system

Abstract

To provide favorable methods and systems for managing and displaying virtual content in a mixed reality system.SOLUTION: Disclosed is an approach for managing and displaying virtual content items in a mixed reality environment on a one-on-one basis independently by respective applications, each virtual content item is rendered by the respective application into a bounded volume referred herein as a "Prism". Each Prism may have characteristics and attributes that allow a universe application to manage and display the Prism in the mixed reality environment, so that the universe application may manage the placement and display of the virtual content in the mixed reality environment by managing the Prism itself.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure generally relates to systems and methods configured to facilitate interactive virtual or augmented reality environments for one or more users. [Background technology]

[0002] Modern computing and display technology has facilitated the development of systems for so-called "virtual reality" (VR), "augmented reality" (AR), and / or "mixed reality" experiences (hereinafter collectively referred to as "mixed reality" and / or "MR"), in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input, while AR or MR scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user, whereby the digital or virtual image (e.g., virtual content) can appear to be part of the real world. However, while MR may integrate virtual content in a contextually meaningful way, AR may not.

[0003] Therefore, there is a need for an approach for managing and displaying virtual content in a mixed reality environment. Summary of the Invention [Means for solving the problem]

[0004] According to some embodiments, instead of each application independently managing and displaying virtual content within the mixed reality environment on a one-at-a-time basis, each virtual content is rendered by its respective application within a bounded volume, which may hereafter be referred to as a “prism.” Each prism may have properties and attributes that enable an application, sometimes referred to as a universe application, to manage and display the prism within the mixed reality environment, whereby the universe application may manage the placement and display of virtual content in the mixed reality environment by managing the prism itself.

[0005] One embodiment is directed to a method for displaying virtual content in a 3D spatial environment, the method including: 1) receiving a request from an application to display virtual content in the 3D spatial environment; 2) creating a prism for managing the display of the virtual content, the prism being a cubic and / or rectangular volume of space configured to confine the boundaries of the virtual content within the boundaries of the prism; 3) receiving the virtual content from the application; 4) rendering the virtual content within the boundaries of the prism and associating the prism with an object in the 3D spatial environment based at least in part on user input; and 6) anchoring the prism within the 3D spatial environment.

[0006] In one or more embodiments, the boundary line of the prism is not displayed. The 3D spatial environment may be the user's physical real-world environment. The prism may be automatically created and have a set of functionality. The set of functionality may include minimum / maximum sizes allowed for the prism and / or aspect ratios for resizing the prism. The set of functionality may include associations between the prism and objects in the 3D spatial environment. The application may render additional virtual content into additional prisms, each virtual content may be rendered into a separate prism.

[0007] In one or more embodiments, a prism does not overlap with other prisms in the 3D spatial environment. A prism may comprise one or more universal features and / or one or more application-specific features to ensure that different applications interact properly with each other, where the one or more universal features and one or more application-specific features are selected from a list of pre-approved options.

[0008] Another embodiment is directed to a display system for displaying virtual content in a 3D spatial environment, the display system may include an augmented reality head-mounted display system and one or more modules for processing data, the one or more modules stored in one or more memories, the one or more modules configured to: 1) receive a request from an application to display virtual content in the 3D spatial environment; 2) create a prism, where the prism is a spatial volume configured to confine the boundary of the virtual content within the boundary of the prism; 3) receive the virtual content from the application; 4) render the virtual content within the boundary of the prism; and 5) associate the prism with an object in the 3D spatial environment.

[0009] In one or more embodiments, the boundary line of the prism is not displayed. The 3D spatial environment may be the user's physical real-world environment. The prism may be automatically created and have a set of functionality. The set of functionality may include minimum / maximum sizes allowed for the prism and / or aspect ratios for resizing the prism. The set of functionality may include associations between the prism and objects in the 3D spatial environment. The application may render additional virtual content into additional prisms, each virtual content may be rendered into a separate prism.

[0010] In one or more embodiments, a prism does not overlap with other prisms in the 3D spatial environment. A prism may comprise one or more universal features and / or one or more application-specific features to ensure that different applications interact properly with each other, where the one or more universal features and one or more application-specific features are selected from a list of pre-approved options.

[0011] Another embodiment is directed to a method for initiating a mixed reality system, the method may include determining a current location of a user, retrieving one or more prisms that were previously deployed at the current location, restoring the one or more prisms at the user's current location, and displaying the restored one or more prisms at the user's current location.

[0012] In one or more embodiments, a prism is a cubic and / or rectangular volume of space within which virtual content from an application is displayed, which may be rendered into two or more prisms. In other words, in some embodiments, a single application may correspond to two or more prisms. In some embodiments, a single application corresponds to a single prism. A prism represents a subtree of a multi-application scene graph for a user's current location. Retrieving one or more prisms previously deployed at a user's current location may include retrieving instance data for the one or more prisms from, for example, an external database, and reconstructing a local prism database with the instance data for the one or more prisms, where the instance data for each prism includes a prism attribute data structure defining the prism, the prism attributes comprising at least one of location, orientation, extent width, extent height, extent depth, anchoring type, and / or anchoring position, and the instance data for each prism includes application-specific attribute key / value pairs comprising state information for virtual content previously rendered in the prism by the application. In some embodiments, the data is stored locally, and no external database is required.

[0013] In one or more embodiments, restoring the one or more prisms includes launching respective applications corresponding to the one or more prisms previously deployed at the current location, creating one or more new prisms corresponding to the one or more previously deployed prisms, and rendering respective virtual content into the one or more new prisms.

[0014] In one or more embodiments, when a user interacts with one or more Prisms, the method further includes updating the user's local Prism database with updated Prism instance data and synchronizing the local Prism database with an external database.

[0015] Some embodiments are directed to a method for managing an application state of virtual content in a mixed reality system, the method may include dividing a 3D volume into a volumetric grid. The method may also include determining a first location of a user within the volumetric grid. The method may further include determining a second location of an application within the 3D volume. The method may also include calculating a distance between the user and the application within the 3D volume. The method may additionally include modifying a state of the application based at least in part on the calculated distance between the user and the application.

[0016] Another embodiment is directed to a method for managing application states of virtual content in a mixed reality system, the method including: recording spatial positions of one or more applications within a volumetric grid, the one or more applications providing content to be displayed inside one or more respective prisms, the volumetric grid corresponding to a rough representation of the physical environment in x, y, and z axes; identifying a cell of the volumetric grid with an application located within the cell, the width of the cell being equal to or greater than a radius of the active zone; determining a distance of a known position of each application within the cell and neighboring cells as a user using a mixed reality device moves within the cell; and modifying the state of the application based at least in part on a distance between the mixed reality device and each application within the cell and neighboring cells.

[0017] In one or more embodiments, the radius of the active zone defines a circular / spherical area around a user using the mixed reality device, and the user may reside at the center of the circle / sphere. In some embodiments, modifying the state of the application may be based, at least in part, on whether the application is occluded by another application. In some embodiments, modifying the state of the application may be based, at least in part, on the user's head pose. The user's head pose is a measurement of the user's head location and / or orientation. The head pose can be used to render a scene to match the user's dynamically changing head location and orientation and provide an increased sense of immersion in the virtual / augmented / composite space. In some embodiments, the head pose may be determined, at least in part, by a head-mounted display system or an inertial measurement unit mounted on the user's head, although other suitable methods may also be used. The distance of each application's known location within a cell may be determined only with respect to the cell in which the user using the mixed reality device resides and neighboring cells.

[0018] In one or more embodiments, a buffer zone may exist around the exterior of the active zone, the buffer zone preventing intermittent or rapid changes to the state of the application.

[0019] Another embodiment is directed to a method for launching an application from a launcher menu for displaying virtual content in a mixed reality system, the method including receiving a request to start the application; creating, by a universe application, a prism for displaying and managing the virtual content, the prism being a stereoscopic display space having a boundary for the application to render the virtual content; starting the application by the prism through a lifecycle service; determining, by the prism, a unique identifier (UID) for the application through a package manager service; registering, by the application, a listener with the universe application; determining, by the universe application, the UID for the application; associating, by the universe application, the listener with the prism; assigning, by the universe application, the prism to the application using the application's listener; and placing the prism within a portion of the 3D viewable space of the mixed reality system.

[0020] Some embodiments are directed to a method for opening and placing an application in an augmented reality environment, the method may include receiving a first user input from a user indicating interest in content. The method may also include launching the application and generating the content. The method may further include creating a mini-display volume of a 3D display volume management unit, where a page preview is displayed in the mini-display volume, and the mini-display volume management unit is created simultaneously with launching the application. The method may further include receiving a second user input indicating movement of the mini-display volume. The method may also include receiving a third user input indicating placement of the mini-display volume at a location in the augmented reality environment and extending the 3D display volume management unit in place of the mini-display volume at the location, where the 3D display volume management unit displays the content fully loaded within the 3D display volume management unit.

[0021] In one or more embodiments, the first user input may be cursor movement over a link on a web page, and the second user input may be selection of the link and movement of the mini-display volume. The mini-display volume may be an initial default size of the 3D display volume management unit. Content may be loaded into the mini-display volume while the user moves and places the mini-display volume. The location may be fixed to an object in the augmented reality environment, the object being the user.

[0022] Some embodiments may be directed to a method of managing virtual content, the method may include receiving the content from a content generation application. The method may also include displaying the content in a 3D spatial environment by a universe application. The method may further include continually managing the display of the content in the 3D spatial environment by the universe application.

[0023] Some embodiments may be directed to a method that includes accessing a scene graph for a scene, the scene graph comprising one or more transform trees, each tree comprising a plurality of nodes. The method may also include adding tags to one or more nodes from the plurality of nodes in the one or more transform trees, the tagged one or more nodes forming a transform group, the tagged one or more nodes comprising a first tagged node and a second tagged node. The method may further include moving the first tagged node, where moving the first tagged node causes the second tagged node to move.

[0024] Another embodiment is directed to a method for displaying virtual content in a 3D shared space, the method may include generating, by a first application, the virtual content in the 3D shared space, and displaying, by a second application, the virtual content generated by the first application, wherein the first application and the second application are different applications.

[0025] Some embodiments may be directed to a method for assigning universal and application-selected features to a prism from a list of pre-approved options for application-based display customization configuration. Another embodiment is directed to a method for displaying virtual content in one or more prisms, where the one or more prisms do not overlap one another. Another embodiment is directed to a method for changing the state of a prism based at least in part on the relative position and location of the prism to a user. Another embodiment is directed to a method for managing content creation in an application and managing content display in a separate application. Another embodiment is directed to a method for opening an application that will provide content into the prism and simultaneously placing the prism in a mixed reality environment.

[0026] Some embodiments may be directed to a method of assigning location, orientation, and range data to a prism for displaying virtual content within the prism, where the virtual content is 3D virtual content.

[0027] In one or more embodiments, the location is the coordinates of the anchoring position of the prism in the mixed reality environment. The range data defines the size of the prism.

[0028] Some embodiments may be directed to a method for pinning a launcher application to a real-world object in a mixed reality environment.

[0029] In one or more embodiments, the pinned launcher application launches the application's content in Prism in the same location as the pinned launcher application.

[0030] Some embodiments may be directed to a method of assigning a behavior type to each prism, the behavior type comprising at least one of world-fixed, billboarding, edge-billboarding, head-following, external sensor-based following, or fading (described in more detail below). Some embodiments are directed to a method of identifying most used content specific to a installed location of a launcher application. Some embodiments are directed to a method of displaying favorite applications by a installed launcher application, the favorite applications based at least in part on a context relative to the location of the installed launcher.

[0031] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims. The present specification also provides, for example, the following items: (Item 1) 1. A method for displaying virtual content in a 3D spatial environment, the method comprising: receiving a request from an application to display virtual content in a 3D spatial environment; creating a prism, the prism being a volume of space configured to confine a boundary of the virtual content within a boundary of the prism; receiving the virtual content from the application; Rendering the virtual content within the boundary of the prism; and associating said prism with an object in said 3D spatial environment; A method comprising: (Item 2) Item 14. The method of item 1, wherein the boundaries of the prisms are not displayed. (Item 3) Item 2. The method of item 1, wherein the 3D spatial environment is a user's physical real-world environment. (Item 4) Item 10. The method of claim 1, wherein the prism is automatically created and has a set of functionality. (Item 5) Item 5. The method of item 4, wherein the set of functionality comprises an association between the prism and the object in the 3D spatial environment. (Item 6) Item 5. The method of item 4, wherein the set of functionality comprises one or more of a minimum size allowed for the prism, a maximum size allowed for the prism, and an aspect ratio for sizing the prism. (Item 7) Item 10. The method of claim 1, wherein the application renders first virtual content in a first prism and second virtual content in a second prism, and the first prism and the second prism are different prisms. (Item 8) 8. The method of claim 7, wherein the prism does not overlap with other prisms in the 3D spatial environment. (Item 9) Item 10. The method of item 1, wherein the prism is placed relative to an object in the 3D spatial environment. (Item 10) 10. The method of claim 9, wherein the object is a user of an augmented reality device. (Item 11) The prism is One or more universal characteristics; One or more application-specific features Equipped with Item 10. The method of item 1, wherein the one or more universal features and the one or more application-specific features are selected from a list of pre-approved options. (Item 12) Item 12. The method of item 11, wherein the one or more universal features ensure that different applications interact with each other in a consistent manner. (Item 13) A display system for displaying virtual content in a 3D spatial environment, said display system comprising means for implementing the method according to any of items 1-12. (Item 14) 1. A method for initiating a mixed reality system, the method comprising: Determining a user's current location; Retrieving one or more prisms previously deployed at the current location; and restoring the one or more prisms at the current location of the user; and displaying the restored one or more prisms at the user's current location; and A method comprising: (Item 15) 1. A method for managing application state of virtual content in a mixed reality system, the method comprising: Dividing a 3D volume into a volumetric grid, determining a first location of a user within the volume grid; determining a second location of the application within the 3D volume; and calculating a distance between the user and the application within the 3D volume; modifying a state of the application based at least in part on the calculated distance between the user and the application; and A method comprising: (Item 16) 1. A method for opening and placing an application in an augmented reality environment, the method comprising: receiving a first user input from a user indicating a request for new content; launching an application and generating said content; creating a mini-display volume of a 3D display volume management unit, wherein a page preview is displayed within the mini-display volume, and the 3D display volume management unit is created simultaneously with the launch of the application; receiving a second user input indicating movement of the mini-display volume; receiving a third user input indicating placement of the mini-display volume at a location in the augmented reality environment; expanding the 3D display volume management unit at the location in place of the mini-display volume; Including, The 3D display volume management unit displays the content loaded entirely within the 3D display volume management unit. (Item 17) 1. A method for managing virtual content, the method comprising: receiving content from a content generation application; displaying said content in a 3D spatial environment via a universe application; continuously managing the display of said content by said universe application in said 3D spatial environment; A method comprising: (Item 18) 1. A method, comprising: accessing a scene graph for a scene, said scene graph comprising one or more transformation trees, each tree comprising a plurality of nodes; adding tags to one or more nodes from a plurality of nodes in the one or more transform trees, the one or more tagged nodes forming a transform group, the one or more tagged nodes comprising a first tagged node and a second tagged node; moving the first tagged node; Including, A method wherein moving the first tagged node causes the second tagged node to move. (Item 19) 1. A method for displaying virtual content in a 3D shared space, the method comprising: generating, by a first application, virtual content in the 3D shared space; displaying, by a second application, the virtual content generated by the first application; Including, The method, wherein the first application and the second application are different applications. [Brief explanation of the drawings]

[0032] The drawings illustrate the design and utility of preferred embodiments of the present disclosure, where like elements are referred to by common reference numerals. In order to better understand how the above and other advantages and objects of the present disclosure are obtained, a more particular description of the present disclosure, briefly described above, will be given by reference to specific embodiments thereof which are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered limiting of its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0033] The drawings use like reference numbers to identify like elements. A letter following a reference number, such as "120a," indicates that the text specifically refers to the element with that particular reference number. A reference number in text without a subsequent letter, such as "120," refers to any or all of the elements in the drawings with that reference number (e.g., "120" in text refers to reference numbers "120a" and / or "120b" in the drawings).

[0034] [Figure 1] FIG. 1 illustrates a system architecture for managing and displaying virtual content in an exemplary user's physical environment and mixed reality system according to some embodiments.

[0035] [Figure 2] FIG. 2 illustrates a system architecture for managing and displaying virtual content in a mixed reality system according to some embodiments.

[0036] [Figure 3] FIG. 3 illustrates a prism / bounding volume according to some embodiments.

[0037] [Figure 4] FIG. 4 illustrates a launcher menu for launching applications and displaying virtual content according to some embodiments.

[0038] [Figure 5] 5A-5B illustrate panel carousel changes according to some embodiments.

[0039] [Figure 6] FIG. 6 illustrates a flowchart for an approach for starting a mixed reality system according to some embodiments.

[0040] [Figure 7] FIG. 7 illustrates a flowchart for an approach for displaying virtual content in a mixed reality environment according to some embodiments.

[0041] [Figure 8] FIG. 8 is a diagram for managing application state according to some embodiments.

[0042] [Figure 9] FIG. 9 shows a flowchart for managing application state for a user location according to some embodiments.

[0043] [Figure 10A] FIG. 10A illustrates a tree node of a scene graph according to some embodiments.

[0044] [Figure 10B] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10C] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10D] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10E] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10F] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10G] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10H] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10I] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10J]10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10K] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10L] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10M] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10N] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10O] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10P] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10Q] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10R] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10S] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10T] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10U] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10V] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10W] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10X] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10Y] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10Z] 10B-10AA illustrate various transform trees and group trees according to some embodiments. [Figure 10AA] 10B-10AA illustrate various transform trees and group trees according to some embodiments.

[0045] [Figure 11] FIG. 11 shows a diagram of a stationary launcher according to some embodiments.

[0046] [Figure 12] FIG. 12 illustrates the types of content that may be displayed while the launcher is in an idle / sleep state according to some embodiments.

[0047] [Figure 13] FIG. 13 illustrates a secondary UI volume according to some embodiments.

[0048] [Figure 14] FIG. 14 illustrates an example of body dynamics according to some embodiments.

[0049] [Figure 15] FIG. 15 illustrates different types of body movements according to some embodiments.

[0050] [Figure 16] FIG. 16 illustrates a flowchart for simultaneously launching and placing applications in a mixed reality environment according to some embodiments.

[0051] [Figure 17]FIG. 17 is a block diagram of an exemplary computing system 1400 suitable for implementing one or more of the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0052] The present disclosure is directed to managing and displaying virtual content in a mixed reality system. Instead of allowing multiple applications to manage and display virtual content in a mixed reality system independently of each other, embodiments of the present disclosure disclose a universe application that uses a 3D window called a prism to manage virtual content (e.g., ways and places) to be displayed and managed in the mixed reality system.

[0053] This disclosure provides a description of an illustrative mixed reality system in which some embodiments of the disclosure may be practiced, followed by a description of one or more embodiments of processes and mechanisms for managing and displaying virtual content in the mixed reality system.

[0054] The following description relates to an exemplary mixed reality system with which the present disclosure may be practiced, however, it should be understood that the present disclosure is also suitable for use in other types of AR and virtual reality (VR) systems, and thus the present disclosure is not limited to only the exemplary system disclosed herein.

[0055] 1 illustrates an exemplary user physical environment and system architecture for managing and displaying virtual content in a mixed reality system. The exemplary environment 100 includes a user landscape 110 as viewed by a user 103 through a head-mounted system 160. The user landscape 110 is a 3D view of the world, and user-placed content can be composited onto the real world. The exemplary environment 100 further includes access to a universe application 130 via a processor 170 operably coupled to a network (not shown). While the processor 170 is shown as a separate and isolated component from the head-mounted system 160, in alternative embodiments, the processor 170 may be integrated with one or more components of the head-mounted system 160 and / or integrated into other system components within the exemplary environment 100, such as, for example, a computing network (not shown) and a network for accessing the external storage device 150. In some embodiments, the processor 170 may not be connected to a network. Processor 170 may be configured with software (e.g., universe application 130) for receiving and processing information such as video, audio, and / or other data (e.g., depth camera data) received from head-mounted system 160, local storage device 137, application 140, a computing network, and / or external storage device 150.

[0056] The universe application 130 may be a 3D window manager, similar to a 2D window manager running on a desktop computer for managing 2D windows displayed on the desktop computer's display screen, for example. However, the universe application 130 (which may hereinafter be referred to as a "universe") manages not only the creation, placement, and display of virtual content 115 in a 3D spatial environment, but also the interactions between multiple pieces of virtual content 115 displayed in the user's landscape 110. The virtual content 115 from applications 140 is presented to the user 103 inside one or more 3D window display management units, such as bounded volumes and / or 3D windows, which may hereinafter be referred to as prisms 113.

[0057] The bounded volume / 3D window / prism 113 may be a rectangular, cubic, cylindrical, or any other shaped volume of space that can be positioned and oriented within space. The prism 113 may be a bounded volume of a stereoscopic display, within which content (e.g., virtual content) is rendered / displayed, and the boundary is not displayed. In some embodiments, the boundary may be displayed. The prism 113 may represent a standard, foundational level of interaction and control of an application's content and its placement. The prism 113 may represent a subtree of a multi-application scene graph, which may be embedded inside a universe or external to but accessible by the universe. A scene graph is a generic data structure commonly used by vector-based graphics, editing applications, and modern gaming software; it arranges the logic, and often (but not necessarily) spatial representation, of a graphical scene. A scene graph may be considered a data structure that defines how content is positioned and transformed relative to each other within that structure. Applications 140 are given an instance of a prism 113 to place content. Applications may render 2D / 3D content into the prism 113 using relative placement algorithms and arbitrary transformations, but the universe may still ultimately be responsible for overall interaction patterns such as content extraction. Multiple applications may render into the universe through prisms 113 with process boundaries separating them. There may be n bounded volumes / prisms 113 per application process; it is explicitly an n:1 relationship in which only one process may run per bounded volume / prism 113 for each application, but there may be m processes running each with their own bounded volume / prism 113.

[0058] The universe operates using a prism / distributed scene graph approach for 2D and / or 3D content. A portion of the universe's scene graph is reserved for each application to render. Each interaction with an application, such as a launcher menu, landscape, or body-centered application zone (all described in more detail below), can occur through a multi-application scene graph. Each application is allocated one to n rectangular prisms, representing a subtree of the scene graph. Prisms are not allocated by client-side applications but instead are created through user interaction inside the universe, for example, when a user opens a new application in the landscape by clicking a button on a controller. In some embodiments, an application may request a prism from the universe, but the request may be denied. In some embodiments, if an application requests a new prism and is granted, the application may only convert the new prism to one of the other prisms.

[0059] The universe places virtual content 115 from applications 140 into objects called prisms 113. Each application process or instance may render its virtual content into its own individual prism 113 or set of prisms. The universe manages the world space (sometimes called the landscape) in which the prisms 113 are displayed. In some embodiments, the universe provides the ability to attach applications to walls and surfaces, place prisms at arbitrary locations in space, register them in the mixed reality system's world database, and / or control the sharing of content among multiple users of the mixed reality system.

[0060] In some embodiments, the purpose of prisms 113 is to provide behavior and control over the rendering and display of content. Similar to 2D displays, where windows may be used to define the location, menu structure, and display of 2D content within a 2D window, in 3D virtual displays, prisms allow the mixed reality system (e.g., universe) to wrap controls around the display of 3D content, for example, related to content location, 3D window behavior, and / or menu structure. For example, control may include at least placing virtual content in a specific location within the user's landscape 110, removing virtual content from the landscape 110, copying virtual content and / or placing copies in a different location, etc. In some embodiments, prisms may be created and destroyed by the user, and only by the user. This may be done explicitly to help control abuse of the interface provided and to help the user maintain control of their content. Additionally, in some embodiments, applications 140 do not know where their volumes are placed within the landscape, only that they exist. In some embodiments, an application may request one or more prisms, and the request may or may not be granted. After a new prism is created, the user may change position, and / or the application may automatically position the new prism relative to the currently existing prisms associated with the application. In some embodiments, each application 140 that utilizes the universe service to render 3D content (e.g., composited 3D content) into the universe process may first be required to register a listener with the universe. This listener may be used to inform the application 140 of the creation and destruction of rendering prisms based on user movement and user interaction with those prisms. The listener is an interface object that receives messages from the inter-process communication system.For example, in the Android operating system, a listener is an object that receives messages through the Android Binder interface. However, any IPC system can be used, so Binder is not always used.

[0061] In some embodiments, a prism may be created from one of the following interactions: (1) a user extracting content from an extractable node (disclosed further below), (2) a user starting an application from a launcher, (3) a user downloading a nearby passable world map tile containing an installed instance of an application that the user has permission to view, (4) a user downloading a nearby passable world map tile containing an object that the passable world object recognizer infrastructure has detected that a given application must render content for, and / or (5) a user triggering a dispatch from another application that must be handled in a different application. In some embodiments, the passable world model allows a user to effectively pass pieces of the user's world (e.g., surroundings, interactions, etc.) to another user.

[0062] Extractable content is content inside a prism (including, but not limited to, icons, 3D icons, words in a text display, and / or images) that can be pulled out of the prism and placed into the landscape using an input device. For example, a prism may display a webpage showing running shoes for sale. To extract the running shoes, the shoes can be selected and "pulled out" using an input device. A new prism is created using a 3D model representing the shoes, and the prism will move out of the original prism and toward the user. Like any other prism, the user can use an input device to move, scale, contract, or rotate the new prism containing the shoes within the 3D space of the landscape. Extractable nodes are nodes in the prism's scene graph that have been tagged as being extractable. In the universe, extracting content means selecting the extractable node and using an input device to pull the content out of the prism. The input to initiate this pulling can be aiming a six-degree-of-freedom pointing device at the extractable content and pulling a trigger on the input device.

[0063] Each user's respective individual mixed reality system (e.g., mixed reality device) captures information as the user moves through or is present in the environment, which the mixed reality system processes to produce a passable world model. Further details regarding passable worlds are described in U.S. patent application Ser. No. 14 / 205,126, filed March 11, 2014, entitled "SYSTEM AND METHOD FOR AUGUMENTED AND VIRTUAL REALITY," previously incorporated by reference. Individual mixed reality systems may communicate or pass the passable world model to a common or shared data collection, referred to as the cloud. Individual mixed reality systems may communicate or pass the passable world model to other users directly or via the cloud. The passable world model provides the ability to efficiently communicate or pass information that essentially encompasses at least the user's field of view. In one embodiment, the system uses the collected 3D points described above as well as pose and orientation information to create the passable world. In some embodiments, a passable world model enables users to integrate content (e.g., virtual and / or physical content) with the real world. A passable world system may include one or more mixed reality systems or mixed reality user devices, which may be connected to a cloud network, a passable world model, a set of object recognizers, and a database (e.g., external database 150). The passable world model may be configured to receive information from the mixed reality user devices and also to transmit data to them over the network. For example, based on input from the users, portions of the passable world may be passed from one user to another. The passable world model may be viewed as a collection of images, points, and other information (e.g., real-world information), based on which the mixed reality system can assemble, update, and build a virtual world on the cloud, effectively passing portions of the virtual world to various users. For example, a set of real-world points collected from the mixed reality user devices may be collected in the passable world model.Various object recognizers may crawl through the passable world model, recognizing objects, tag images, etc., and attach semantic information to the objects. The passable world model may use a database to build knowledge of its world, attach semantic information, and store data associated with the passable world.

[0064] For a prism that is visible to the user but whose controlling application is not currently installed, the universe may render a temporary placeholder for that application that, when interacted with, redirects the user to the application store page for that application.

[0065] In some embodiments, a prism may be destroyed upon similar interactions: (1) the user moves far enough away from a passable world map tile that the installed instance of the application is unloaded (i.e., removed) from volatile memory, (2) the user destroys the installed instance of the application, and / or (3) the application requests that the prism be closed.

[0066] In some embodiments, if prisms for an application are not visible and / or loaded, the processes associated with those prisms may be paused or terminated. When the installed prisms for that application become visible again, the processes may resume. Prisms may also be hidden, but in some embodiments, this may occur only at the request of the universe and the user. In some embodiments, multiple prisms may be installed in the same exact location. In such embodiments, the universe may manage rendering by showing only one instance of an installed prism in a location at a time and hiding the visibility of the prism (and its associated content) until a user interaction is detected, such as the user "swiping" to the next visible element (e.g., a prism) in that location.

[0067] In some embodiments, each prism 113 may be exposed to applications 140 via a volume listener interface, with a way to access prism 113 attributes and register content in a scene graph subtree for shared resources such as meshes, textures, and animations.

[0068] In some embodiments, because the application 140 does not know where a given prism 113 will be placed in 3D space, the volume listener interface may provide accessor methods to a set of hints that help define where a given prism resides in the universe, such as hand-centered, fixed in landscape, or body-centered. These attributes additionally specify the prism's expected behavior and can be controlled in a limited manner by either the user, the application 140, or the universe. A given prism can be positioned relative to other prisms owned by the application. An application can specify that prisms from that application should snap together (that is, both sides of their bounding volumes touch) while being placed. Additionally, prisms may provide an API for key / value data storage. Some of these key / value pairs are writable only by authorized applications.

[0069] In some embodiments, application 140 is a client software application that provides content to be displayed to user 103 in user's landscape 110. For example, application 140 may be a video streaming application, where video data may be streamed to the user to be displayed on a 2D planar surface. As another example, application 140 may be the Halcyon application, which provides 3D imaging of physical objects that may represent past periods of calm, happiness, and rest for the user. Application 140 provides content that the user may want to include in user's landscape 110. Universe, via Prism 113, manages the placement and management of content generated by application 140.

[0070] When a non-immersive application is executed / launched in the user's landscape 110, its content (e.g., virtual content) is rendered inside the prism 113. A non-immersive application may be an application that is capable of launching and / or displaying content simultaneously with one or more other applications in a shared 3D environment. Although the virtual content may be contained in the prism, the user may still interact with the virtual content, for example, by hovering over an object, clicking on it, etc. The prism 113 may define the boundaries of the displayed content of the bounding application 140, so that different applications 140 do not interfere with each other or with other objects in the user's landscape 110. The prism 113 may also provide a useful abstraction for suspending, pausing, and / or minimizing virtual content from applications 140 that are outside the user's field of view or too far away from the user.

[0071] Prisms 113 can be anchored / attached / pinned to various objects within the user's landscape 110, including snapping or anchoring to another prism. For example, a prism 113a displaying virtual content 115 (e.g., video 115a from a video streaming application) can be anchored to a vertical wall 117a. As another example, a prism 113b displaying a 3D tree 115b from the Halcyon application is shown in FIG. 1 as being anchored to a table 117b. Additionally, prisms 113 can be anchored relative to the user 103 (e.g., body-centered), where the prism 113 displaying the virtual content 115 can be anchored to the user's body such that as the user's body moves, the prism 113 moves relative to the user's body movements. Body-centered content can be application content such as a plane, mesh, etc. that follows the user and remains positionally aligned with the user. For example, a small dialog box that follows the user but resides relative to the user's spine rather than the landscape 110. Additionally, prism 113 may be anchored to a virtual object, such as a virtual display monitor, displayed within user landscape 110. Prism 113 may be anchored in different ways, as disclosed below.

[0072] The universe may include a local database 137 for storing attributes and characteristics of prisms 113 for the user. The stored prism information may include prisms activated by the user within the user's landscape 110. The local database 137 may be operatively coupled to an external database 150, which may reside in the cloud or an external storage facility. The external database 150 may be a persistent database that maintains information about the user's and other users' mixed reality environments.

[0073] For example, when a user launches a new application to display virtual content in the user's physical environment, local database 137 may store information corresponding to prisms created by universe and installed in specific locations, and application 140 may render the content into prism 113 to be displayed within the user's landscape 110. The information corresponding to prism 113, virtual content 115, and application 140 stored in local database 137 may be synchronized with external database 150 for persistent storage.

[0074] In some embodiments, persistent storage may be important because data stored in local database 137 may be erased, deleted, or become non-persistent when the mixed reality system is turned off. Thus, when a user turns on the mixed reality system, the universe may synchronize with external database 150 and retrieve an instance of local database 137 corresponding to user 103 and the user's landscape 110 prior to the mixed reality system being turned off. Local database 137 may be an instance of external database 150, which includes information related to user 103 and the user's current environment. External database 150 may additionally store local database instances for other users, multiple users, the same user over time, and / or other environments. External database 150 may include information used to manage and share virtual content among multiple users of the mixed reality system, while local database 137 stores and maintains information corresponding to user 103.

[0075] The universe may create a prism 113 for an application 140 whenever the application 140 needs to render virtual content 115 on the user's landscape 110. In some embodiments, the prism 113 created by the universe allows the application 140 to focus on rendering the virtual content for display, while the universe focuses on creating and managing the placement and display of the prism 113 with the virtual content 115 displayed within the prism's boundaries by the application 140.

[0076] Each piece of virtual content 115 rendered by application 140 displayed within user landscape 110 may be displayed within a single prism 113. For example, if application 140 needs to render two pieces of virtual content (e.g., 115a and 115b) to be displayed within user landscape 110, application 140 may render two pieces of virtual content 115a and 115b. Because virtual content 115 includes only the rendered virtual content, universe may create prisms 113a and 113b to correspond to each piece of virtual content 115a and 115b, respectively. Prism 113 may include 3D window management attributes and properties for virtual content 115, allowing universe to manage the virtual content 115 inside prism 113 and the placement and display of prism 113 in user landscape 110.

[0077] The universe may be the first application visible to the user 103 when the user 103 turns on the mixed reality device. The universe may be responsible for at least (1) rendering the user's world landscape, (2) 2D window management and 3D window (e.g., prism) management for planar applications, (3) displaying and executing application launcher menus, (4) allowing the user to place virtual content into the user's landscape 110, and / or (5) managing different states of the display of prisms 113 within the user's landscape 110.

[0078] Head-mounted system 160 may be a mixed reality head-mounted system that includes a display system (e.g., a user interface) positioned in front of the eyes of user 103, speakers coupled to the head-mounted system and positioned adjacent the user's ear canals, a user sensing system, an environmental sensing system, and a processor (all not shown). Head-mounted system 160 presents user 103 with the display system (e.g., a user interface) for interacting with and experiencing the digital world. Such interactions may involve the user and the digital world, one or more other users interfacing with representative environment 100, and objects within the digital and physical worlds.

[0079] The user interface may include viewing, selecting, positioning, and managing virtual content via user input through the user interface. The user interface may be at least one or a combination of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, an audio input device, a smartphone, a tablet, or a head-mounted system 160. A haptic interface device is a device that allows a human to interact with a computer through bodily senses and movement. Haptic refers to a type of human-computer interaction technology that involves tactile feedback or other bodily senses to perform an action or process on a computing device.

[0080] An example of a haptic controller may be a totem (not shown). In some embodiments, the totem is a handheld controller that tracks its position and orientation relative to the headset 160. In this example, the totem may be a six-degree-of-freedom (6DOF) controller that allows a user to move the prism in elevation and azimuth (on the spherical shell) by moving the totem up or down. In some embodiments, to move an object closer or farther away, a user may use a joystick on the totem to "push" or "pull" the prism, or simply move the totem forward or backward. This may have the effect of changing the radius of the shell. In some embodiments, two buttons on the totem may expand or contract the prism. In some embodiments, rotating the totem itself may rotate the prism. Other totem operations and configurations may be used and should not be limited to the embodiments described above.

[0081] The user sensing system may include one or more sensors 162 operable to detect certain characteristics, properties, or information related to the user 103 wearing the head-mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical properties / measurements of the user 103, such as one or more of: pupil constriction / dilation, angulation / positioning of each pupil, sphericity, eye shape (as eye shape changes over time), and other anatomical data. This data may provide or be used to calculate information (e.g., the user's visual focus) that can be used by the head-mounted system 160 to enhance the user's viewing experience.

[0082] The environmental sensing system may include one or more sensors 164 for acquiring data from the user's landscape 110. Objects or information detected by the sensors 164 may be provided as input to the head-mounted system 160. In some embodiments, this input may represent a user interaction with the virtual world. For example, a user (e.g., user 103) viewing a virtual keyboard on a desk (e.g., table 188) may gesture with their fingers as if they were typing on the virtual keyboard. Finger movement motions may be captured by the sensors 164 and provided as input to the head-mounted system 160, where the input may be used to change the virtual world or create new virtual objects.

[0083] Sensor 164 may, for example, generally include an outward-facing camera or scanner for capturing and interpreting scene information, for example, through continuously and / or intermittently projected infrared structured light. The environmental sensing system may be used to map one or more elements of user landscape 110 around user 103 by detecting and registering one or more elements from the local environment, including static objects, dynamic objects, people, gestures, various lighting, atmospheric and acoustic conditions, etc. Thus, in some embodiments, the environmental sensing system may include image-based 3D reconstruction software embedded in a local computing system (e.g., processor 170) and operable to digitally reconstruct one or more objects or information detected by sensor 164.

[0084] In some embodiments, the environmental sensing system provides one or more of motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, acoustic source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optical enhancement sensors. It should be understood that the environmental sensing system may include other components besides those discussed above.

[0085] As mentioned above, processor 170 may, in some embodiments, be integrated with other components of the system of representative environment 100, integrated with other components of head-mounted system 160, or may be a separate device (wearable or separate from user 103) as shown in FIG. 1 . Processor 170 may be connected to various components of head-mounted system 160 through a physical wired connection or through a wireless connection, such as, for example, a mobile network connection (including cellular telephone and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. Processor 170 may include memory modules, integrated and / or additional graphics processing units, wireless and / or wired Internet connectivity, and codecs and / or firmware capable of converting data from sources (e.g., computing networks and user and environmental sensing systems from head-mounted system 160) into image and audio data, which may be presented to user 103 via a user interface (not shown).

[0086] Processor 170 handles data processing for various components of head-mounted system 160 as well as data exchange between head-mounted system 160 and software applications such as Universe, external database 150, etc. For example, processor 170 may be used to buffer and process data streaming between user 103 and a computing network containing software applications, thereby resulting in a smooth, continuous, and high-fidelity user experience. Processor 170 may be configured to execute a set of program code instructions. Processor 170 may include a memory for retaining a set of program code instructions, wherein the set of program code instructions comprises program code for displaying virtual content within a portion of the available 3D viewable space by displaying the virtual content within a stereoscopic display space, where the boundaries of the stereoscopic display space are not displayed. In some embodiments, the processor may be two or more operably coupled processors.

[0087] In some embodiments, the mixed reality system can be configured to assign universal features and application-selected / application-specific features to prisms from a list of pre-approved options for application-specific display customization configuration. For example, universal features ensure that different applications interact well together. Some examples of universal features can include maximum / minimum size, non-overlapping prisms (excluding temporary overlaps from collision behavior), hiding content outside the prism's boundaries, and requiring an application to require permission from the user if the application wants to access sensors or senseable information. Application-selected / application-specific features result in an optimized application experience. Application-selected / application-specific features may include maximum / minimum size (within system-imposed limits), default size (within system-imposed limits), type of body movement (e.g., unfixed / world-fixed, billboarding, edge billboarding, following / passive head fixing, following based on external sensors, fading, as discussed below), child prism generation location, child head pose highlighting, child prism relationship behavior, surface behavior, independent transform control, resize vs. scale, idle timeout, collision behavior, permission / password for accessing the application, etc. In another embodiment, the mixed reality system may be configured to display virtual content in one or more prisms, which in some embodiments do not overlap with each other. In some embodiments, one or more prisms may overlap to provide specific interactions. In some embodiments, one or more prisms may overlap, but only with other prisms from the same application. In another embodiment, the mixed reality system may be configured to change the state of a prism based at least in part on the relative position and location of the prism to the user. In another embodiment, the mixed reality system may be configured to manage content creation in an application and content display in a separate application.In another embodiment, the mixed reality system may be configured to open an application that will provide content into a prism and simultaneously place the prism in the mixed reality environment.

[0088] In some embodiments, the mixed reality system may be configured to assign location, orientation, and range data to the prisms to display virtual content within the prisms, where the virtual content is 3D virtual content. In some embodiments, the mixed reality system may be configured to pin the launcher application to a real-world object within the mixed reality environment. In some embodiments, the mixed reality system may be configured to assign a behavior type to each prism, where the behavior type comprises at least one of world pinning, billboarding, edge billboarding, follow head pinning, follow based on external sensors, or fading (described in more detail below). In some embodiments, the mixed reality system may be configured to identify the most used content or applications specific to the installed location of the launcher application and, as a result, sort the applications, for example, from most frequently used to least used. In another embodiment, the mixed reality system may be configured to display favorite applications in the installed launcher application, where the favorite applications are based, at least in part, on context related to the installed launcher's location.

[0089] 2 illustrates a system architecture for managing and displaying virtual content in a mixed reality system according to some embodiments. System 200 includes universe 130, applications 140, icon grid application 260, status bar app 270, social panel app 280, and store panel app 290. These applications may represent a base level of applications on system 200; however, in some embodiments, more or fewer applications may be part of system 200.

[0090] As discussed above in Figure 1, the universe may be viewed as a 3D window (e.g., prism) manager similar to a 2D window manager that manages 2D windows in a traditional computer desktop system, etc. Figure 2 may provide further details of the universe from Figure 1, where the universe application 130 may include a universe server 205, a loader volume 210, a secondary UI volume 220, a universe client 225, a launcher application 230, and the universe server 205. The universe server 205 may be the processing thread of the universe in a multi-threaded processing environment for multi-parallel processing.

[0091] Loader volumes 210 are placeholder volumes that are displayed to the user while universe is creating a prism for displaying virtual content within the user's landscape 110. For example, when a user selects an application to display in a particular location within the user's landscape 110 (e.g., on a vertical wall of the user's landscape 110), universe may display loader volume 210 as a placeholder volume with a default icon while universe sets up the prism and starts the application for rendering the virtual content into the prism, indicating to the user that universe is setting up the prism for display. Once the application has finished rendering the virtual content into the prism for display in the user's landscape, loader volume 210 is replaced with the actual prism containing the rendered virtual content.

[0092] In some embodiments, while Universe is launching an application to display the virtual content, the user 103 may move the loader volume 210 to a different desired location. In some embodiments, the user may move the loader volume 210 to a location different from the location of the originally selected loader volume / prism. Once Universe has finished creating the prism and the application is rendering the virtual content into the prism, Universe may replace the loader volume 210 with a prism that displays the virtual content each time the user places the loader volume 210.

[0093] A secondary UI volume 220 is another prism that may be created when a prism 113 (e.g., its “parent prism”) is created. The secondary UI volume 220 provides a universal interface for the prism for the user. For example, the secondary UI volume 220 may be considered window dressing because the secondary UI volume 220 provides a mechanism for managing the prism (e.g., closing / removing, sharing, following, taking screenshots of the prism's contents, etc.). When a prism is created, if the prism is not part of a launcher (a launcher application may not have a secondary UI volume), a secondary UI volume 220 may be created for the prism. The secondary UI volume 220 provides a space / volume for displaying graphical user interface icons for the user to interact with and manage the prism (e.g., closing / removing, sharing, following, taking screenshots, etc.). The secondary UI volume 220 may be associated with and grouped with the parent prism. The life of a secondary UI volume 220 ends when the life of the parent prism to which it is associated ends.

[0094] In some embodiments, the secondary UI volume 220 may have at least three states: (1) display nothing when the parent prism is out of focus; (2) display the component's “visible name” when the parent prism is in focus; and (3) display a “carousel” of application menu option icons when a specific user interaction is detected (e.g., holding the home button on a handheld controller (e.g., a totem or other suitable user interaction controller) for a certain number of seconds), where the carousel displays a collection of icons, one of which may be a large “X” icon for closing the prism. In some embodiments, the secondary UI volume 220 receives input through its parent prism. In other words, the parent prism may determine whether the secondary UI volume 220 is displaying its carousel, and if so, the parent prism redirects user input to the secondary UI. The carousel of the secondary UI volume 220 is disclosed below.

[0095] In some embodiments, the launcher can be the default "home" menu for the mixed reality system. The launcher can bring together multiple panels of content alongside the system status bar. Each panel can represent a different content type. Applications can be pulled from the launcher and pinned into the landscape for quick return. The launcher itself can also be placed into the landscape for location-specific customization and / or quick access.

[0096] The launcher 230 provides users with the ability to launch new applications into their landscape 110. The launcher 230 may be an application consisting of a series of body-centered prisms called panels. The panels may be vertically and horizontally scrollable, and the user may switch between panels using, for example, a swipe motion. In some embodiments, one panel (e.g., the center panel) may be visible at a time, with two neighboring panels visible on its sides as placeholder panels. When the user swipes to the next panel, the placeholder panel may expand to reveal the full panel. The panels may include an icon grid application 260, a social panel 280, and a store panel 290. In some embodiments, when the user swipes to the next panel, the panels themselves do not move or change, but instead, content (e.g., icons) in different panels may be animated in and out of the center panel (e.g., the active panel). Additionally, applications can be pulled from the launcher 230 and pinned into the user's landscape 110 for location-specific customization, as discussed further below.

[0097] In some embodiments, applications 140 may communicate with the universe through a centralized rendering service client 250 on each application 140. The centralized rendering service client 250 may communicate with a universe server 205 in universe 130. The centralized rendering service client 250 may be a client service of a centralized rendering system that allows applications 140 and other applications that generate content to communicate with the universe through the universe server 205 for display in the user's landscape.

[0098] The universe server 205 may comprise the services of a centralized rendering system that allows the universe to communicate with applications that provide universe content for display within a user's landscape. In some embodiments, the communication may comprise more than rendering data, for example, input data, requesting security permissions, requesting to show or hide a virtual keyboard, etc.

[0099] In some embodiments, a centralized rendering system may be a system dedicated to hardware and software resources for receiving graphical data from multiple applications for display on a single display (e.g., in a user's landscape in a mixed reality system). The centralized rendering system combines graphical data from multiple applications 140 into a “centralized” data structure, such as a scene graph, which can be used to render a scene on a display that reflects the graphical data from the multiple applications in a realistic and efficient manner. To achieve a centralized rendering system, in some embodiments, applications may make changes to a local representation of a prism, called a client prism (e.g., client prism 215 from FIG. 2 ). These changes may then be transmitted to universe server 205 and stored in the server prism. The centralized rendering system may then render the updated data in the server prism. The centralized rendering system may hereafter be referred to as the “Cali” or “Kali” system. Universe may be considered an extended version of the Cali server, for example, since universe may manage prisms in the real world.

[0100] In some embodiments, each application 140 that creates virtual content 115 for a universe communicates with the centralized rendering system and the universe via a centralized rendering service client 250 (hereinafter, referred to as a “Cali client”) installed in each respective application 140. Further information may be disclosed in related application Ser. No. 62 / 479,134, entitled “CENTRALIZED RENDERING,” filed March 30, 2017, and incorporated herein by reference in its entirety. The centralized rendering system improves the user's experience by ensuring that virtual content from multiple different applications is properly analyzed and processed, and, if necessary, that the virtual content is displayed to the user in a realistic manner. In some embodiments, a universe is an instance of a Cali server with additional functionality, such as managing prisms. In some embodiments, a client prism is an instance of a Cali client volume, and a server prism is an instance of a Cali server volume, with additional functionality such as providing an app options display, displaying a loader volume while a prism loads its content, the ability to collide with other prisms, and the ability to be part of a transformation tree.

[0101] Client prism 215a and client prism 215b comprise virtual content generated by application 140, which was sent by Cali client 250a to universe server 205 for display in the user's landscape. In some embodiments, when application 140 makes changes to virtual content 115a and 115b, the changes to the virtual content are communicated from client prism 215 to universe server 205, and the information is stored inside the universe in a corresponding server prism data structure 113. In some embodiments, application 140 does not know where in the user's landscape virtual content 115a is displayed. The universe may manage the display location of virtual content 115a (e.g., virtual content 115a after it has been processed by a centralized rendering system) through a corresponding server prism 113a associated with client prism 215a. Application 140 may request a new prism by accessing universe server 205. In some embodiments, the universe server 205 may be a software module in the universe that communicates with a centralized rendering service client 250 from applications that provide virtual content for display within the user's landscape 110. For example, when a user launches an application and wants virtual content from the application to be displayed within the user's landscape, the application may provide the virtual content from the application to the universe centralized rendering service on the universe via the centralized rendering service client to be displayed within a prism that may be anchored within the user's landscape.

[0102] In some embodiments, the icon grid application 260 may include a recent applications section (not shown) and / or a general applications section (not shown). The general applications section includes an icon representing each application installed on the mixed reality system. The general applications section may initially be populated with a call to a package manager (not shown) to determine a list of installed packages. An icon is added for each application in each package. As the package manager notifies the universe of package installations and uninstallations, the icon grid application 260 adjusts its icons accordingly. The package manager service manages the installation of applications and maintains information about those applications (such as their names, icon graphics, security permissions, executable files, and data files).

[0103] The recent icon section is first reconstructed from the log on disk and may then be updated by calls from other services. Package names may be logged to disk when the lifecycle service notifies the launcher of an application start event and when the package manager notifies the launcher of a package uninstall event. The user may interact with the icon grid application 260 by selecting an icon to launch or extracting an icon to place in the landscape.

[0104] The lifecycle service may be a centralized service that manages the processes of starting, stopping, sleeping, and waking up applications. The lifecycle service also knows when an application terminates unexpectedly (crashes). When any of these events occur, listeners for the service are notified, and the universe is one of the listeners. The universe accesses this service to start, stop, sleep, and wake up applications. In some embodiments, the lifecycle service provides an application programming interface (API) for controlling the lifecycle of application processes that launch within the mixed reality system. The lifecycle service may spawn new processes to launch application binaries with a set of permissions and call the API on predefined interfaces implemented by the applications to control their lifecycle. The lifecycle service also provides a listener interface through which other modules can record applications being started / stopped / paused / resume. The lifecycle service may be a program separate from the launcher or the universe. In some embodiments, the lifecycle service may be middleware.

[0105] 2, icon grid application 260 comprises centralized rendering service client 250b and client prism 215c. As discussed above, in some embodiments, applications that display content within a user's landscape may send their content to the universe via centralized rendering service client 250, which communicates with universe server 205. Here, icon grid application 260, which provides icons for applications installed on the mixed reality system for a launcher menu, is similar to any other application that provides content for display in the user's landscape. However, in some embodiments, an icon within the icon grid application, when selected by a user, may instruct universe to launch and start a new application, at which point the new application may request universe to create a new prism (e.g., through universe server 205) so that the application can provide content for display in the new prism. If the application is already running, universe may request the application to open the new prism.

[0106] The status bar application 270 provides status indicators for the mixed reality system. The status indicators and the status bar application 270 may not be user adjustable. The status indicators may be initially populated by querying a first service for operating and maintaining WiFi services, a second service for maintaining Bluetooth services, and a third service for status. When these services notify the status bar application 270 of updated status, the status bar may adjust accordingly. The status bar provides users with quick-glance information that they can react to quickly and efficiently from anywhere in the system. In some embodiments, the status bar may be displayed above the launcher. Four main sections in the status bar may be (1) global search, (2) notifications, (3) quick settings, and (4) power. Additional temporary sections, such as music, phone, and sharing, may be added to the status bar as needed.

[0107] When the user is in the launcher menu, the status bar is reduced to glance icons. When the user swipes to the top, it may trigger animation and the status bar may expand. The status bar may stay above the launcher while the user may swipe left and right through the launcher panel. When the status bar is highlighted, it may expand and animate forward. A sub-selection highlight may appear on the left by default, for example, above global search. If there is another section with more urgent content (e.g., recent notifications, low battery, etc.), the sub-selection highlight may appear above that section instead.

[0108] The social panel application 280 may consist of a set of contacts with which a user may interact. The social panel may be initially populated with a call to a contacts service for available contacts. Each contact may be added to the social panel and displayed to the user as an icon. As the social panel application 280 receives new contact, updated contact, and removed contact events, the social panel application 280 may adjust its contact information accordingly. A user may interact with a contact icon by clicking on it to pop up an options menu with various contact providers available. When the user selects a provider, a launcher application may launch the associated application with the contact's information.

[0109] The store panel application 290 may allow a user to search for, download, and install applications 140 for the mixed reality system. When a user requests to download and install an application, the launcher application 230 may verify the user's identity using an identity verification service (not shown) and then install the application using a package manager. Lifecycle services may be invoked when a user starts an application from a panel. In some embodiments, each panel in the launcher may function as a separate application instead of as one launcher application.

[0110] In some embodiments, the universe client 225 renders specific content for the universe. The universe server 205 does not render third-party applications. This is because content within prisms can only be rendered by the universe client 225, not the universe server 205. Therefore, to render infinity prisms, loader volume / prisms, and / or secondary UI prisms, the work may need to be delegated to the universe client 225 to render those specific types of content for the server. Infinity prisms may be used by the universe to render additional graphics around a prism, for example, when two prisms collide. Infinity prisms are discussed further below. In loader prisms and secondary UI prisms, specific communication may exist between the universe server 205 and the universe client 225 to coordinate certain functionality. For example, the universe server 205 may be told that an application has finished loading. The universe server 205 may then notify the client-side loader prism that it is now loading the application. The loader prism would need to react to the event that the application has finished loading by showing the animation. When the client-side loader prism has finished showing the animation, the loader prism may notify the universe server 205 that it has finished animating. The universe server 205 may then react to the notification that the loader prism has finished animating by dismissing the loader prism, discarding the loader prism, and displaying the app prism with rendered animation in place of the loader prism. The disclosed subject matter is merely one example of how the universe client 225 may function. Those skilled in the art will understand that there may be other examples of when the universe client 225 may assist the universe 130.

[0111] Figure 3 shows an example bounding volume / prism according to some embodiments. Application content is presented to a user inside one or more bounding volumes called prisms. As discussed above, when a non-immersive application runs within a mixed reality system, its content is rendered inside a prism. Prism attributes and properties allow the universe to consistently manage prisms within the user's landscape.

[0112] The volumetric space of the prism 113 may have a clear and definitive boundary, as shown by the dashed line in FIG. 3 . The boundary provides a bounding volume for the virtual content 115 to be displayed only within the bounds of the prism 113. The prism boundary prevents content from an application displayed within the prism from overflowing, or spilling, out of the prism and into the user's landscape. The prism boundary 113 may not be visible to the user when the user views the virtual content 115 displayed within the prism 113. This is an important feature because, in order to maintain a realistic display of 3D content within the user's landscape, it is important not to show the prism boundary that bounds the virtual content 115. Those skilled in the art will understand the importance of not showing the prism boundary that envelops the virtual content 115 so that the virtual content is displayed within the user's landscape in a more realistic manner. In contrast to 3D windows, the borders and boundaries of 2D windows are generally displayed so that a user of a computer displaying 2D windows can clearly distinguish content within one 2D window from content from another 2D window. However, in some embodiments, it may be advantageous to at least temporarily display the prism boundaries, for example, to help troubleshoot problems with one or more applications.

[0113] An application is given an instance of a prism 113 by the universe in which to place content. Applications may render 2D and / or 3D content into the prism 113 using relative placement algorithms and / or arbitrary transformations, but the universe is still ultimately responsible for managing the overall interaction patterns, such as content extraction. Multiple applications may render process boundaries separating prisms into the universe through the prism 113.

[0114] Each prism distributed within a universe has an associated set of key / value attributes that can be adjusted to determine various bits of behavior or convey information about why a given prism exists. Some attributes are read-only for normal applications, but for applications with private APIs, these attributes are writeable. Prisms 113 may include prism attributes 310, application-specific attributes 320, and virtual content 115. In addition, some prisms 113 include a secondary UI volume 330 to provide users with additional prism management options. However, in some embodiments, a prism may not have a secondary UI volume 330 because, for example, these other types of prisms (e.g., launcher menu prisms) may not require the features provided by the secondary UI volume 330. Like a prism's border, the secondary UI volume 330 may not be visible to the user. When a user wants to make changes to the prism, the user can initiate a request to display an application options menu, which displays the UI controls for the prism within the volume space of the secondary UI volume.

[0115] Prisms may require different attributes to provide appropriate feedback and behavior for their content depending on the application they contain. Application developers can select from several pre-programmed options for their Prisms when they create their applications, so their content can be properly presented based on their preferences. Below are some examples of these options:

[0116] Prism attributes 310 at least partially define a prism and allow universe to manage and maintain prisms within a user's landscape. For example, prism attributes 310 may include one or more of: default size, maximum size, minimum size, anchoring / placement type (e.g., options for billboard display, etc.), behavior of a given prism relative to anchoring type, anchoring location, child prism creation location, child head pose highlight, surface behavior, independent transform control, resize vs. scale indicator, idle timeout variable, etc. Prism attributes 310 provide universe with the ability to track and manage every single prism within a user's landscape. Having a single application that manages the virtual content displayed within a user's landscape ensures that content displayed within a user's landscape is displayed in a consistent and reliable manner. Some of prism attributes 310 are disclosed further below.

[0117] Maximum, minimum, and default size: Applications can have upper and lower bounds defined by the application developer (optionally with additional limits from the universe). In addition, the application developer can have a default size when the application is first launched.

[0118] Option to billboard during movement sequences: It makes sense for certain objects (e.g., flat content) to billboard towards the user during movement sequences to facilitate visibility and less management. For example, some content displayed on a planar surface may be positioned relative to a particular location and / or object, but its orientation is automatically calculated so that the content displayed on the planar surface always faces the direction of the user viewing the content displayed on the planar surface. Other optional body movement behaviors may also be added to this.

[0119] Child Prism Creation Location: Prisms can create children to create flexible layouts. Application developers should be able to determine the responsiveness of where children can be created relative to a parent prism.

[0120] Child Head Pose Highlights: Applications may be able to choose whether head pose highlights on child prisms can be treated as separate highlights or whether to continue highlighting all child / parent prisms as one unit.

[0121] Child Prism Relationship Behavior: Prisms determine whether their child prisms can be anchored to them in translation, rotation, and scale, and can choose whether child prisms will end up with the main prism.

[0122] Surface Behavior: Prisms may snap to a surface and query that surface to determine if they want to change size / scale. If the surface has voids, the prisms may resize to fit all or a percentage of the surface and into the user's field of view (FOV).

[0123] Independent Transform Control: An application may request independent control of its translation, rotation, and scale. This may allow the application to move and transform itself.

[0124] Resizing vs. Scaling: Some applications may choose to resize their boundaries instead of scaling only their content. This may accommodate more content being displayed within their boundaries. This may function like a traditional computer 2D window.

[0125] Idle timeout: Applications may be able to choose how long it takes before entering their idle state. This may handle situations where an application may want to continue playing content even when out of view. For example, an application that displays live video may want to continue displaying content and playing audio even if the user temporarily looks away.

[0126] Application-specific attributes 320 may be a list of key / value pairs that store application-specific state information for each Prism. The list of key / value pairs is specific to the application, and the key / value pairs provide state information for the application's content being displayed or rendered within the Prism. The list of key / value pairs may be different for each Prism depending on the application rendering within the Prism. For example, if the application is a video streaming application, some key / value pairs may include the video name, the maximum viewing time for the video, the aspect ratio for displaying the video, etc.

[0127] Both prism attributes 310 and application-specific attributes 320 may be stored for each prism in a data structure in local database 137. The prism data is continually updated while the user operates the mixed reality system and interacts with the prisms. As discussed above, the prism instance data in local database 137 may be persisted by synchronizing with external database 150 on a periodic basis. In some embodiments, local database 137 and external database 150 may be synchronized in near real time.

[0128] When a user launches an application in the universe, the user can pull out a prism from a launcher menu and place the resulting volume in space. Other methods of launching an application, such as clicking an application icon, can also be used. In some embodiments, the user can move the prism in altitude and azimuth (on the spherical shell) by moving a controller / input device (e.g., a totem) up or down. To move an object closer or farther away, the user can use a joystick on the totem to "push" or "pull" the prism, or slide their finger over a touch-sensitive portion of the totem. This has the effect of changing the radius of the shell. In some embodiments, two buttons on the totem can expand or contract the prism. Finally, rotating the totem itself can rotate the prism. This assumes that the totem may have six degrees of freedom (DOF). This is consistent with the types of controls used in VR painting applications, for example, but the totem can be any suitable user input device.

[0129] In some embodiments, prisms may not allow themselves to be placed in such a way that they fully or partially intersect with other prisms. Prisms may not intersect at all or not be in / actively display in the exact same location (anchor point), with the exception that prisms may only slightly overlap for physics purposes, as discussed below. If two or more prisms are placed in the exact same location, the active application may be displayed and other applications anchored in the exact same location may be hidden. The user may know that multiple applications are located by, for example, dots displayed within the volume. For example, if three prisms / applications are in a particular spot, there may be three dots. If the user views application #2 of the three, the second dot may be brightened and the other dots may be dimmed. The user may then swipe or scroll through different applications. The graphics may switch, and the dots may be updated (e.g., by brightening the active dot) to indicate the currently active application.

[0130] In some embodiments, several prisms may be co-located in the same anchoring location. At first glance, that may seem odd. With all the 3D space available to place applications in the user's landscape, why place them in the same spot? In fact, it makes perfect sense. For example, a user's favorite place to play virtual board games may be on the kitchen table. In the morning, the user may prefer to play "Ticket To Ride" while eating breakfast. However, when the user gets home from work, the user may prefer to play "Risk" against the computer. The user may have multiple board games located in the same spot and switch between them as needed.

[0131] In some embodiments, a prism may be placed at an arbitrary location in space. In this case, the prism may be anchored by the center point of a cubic / rectangular volume. However, if the prism is moved near a horizontal surface in the landscape (e.g., during placement), the prism may attempt to snap to the surface. The anchor point may then be the center of the prism's bottom plane. Similarly, if the prism is moved toward a vertical surface (e.g., a wall), it may attempt to snap to it, and the anchor point may be the side of the prism that is next to the vertical surface.

[0132] The purpose of an anchor point may be to position a prism so that it does not interpenetrate with the surface it is anchored to. An anchor point may also move with the object it is anchored to. When multiple prisms share the same location, that location may be the anchor point, not the center point of their respective volumes. An application does not know where they are located, and does not need to know, but it may query each of those prisms to see how each is anchored. An application may also specify which anchor types are valid. For example, it would not make sense to anchor a Halcyon to a vertical surface.

[0133] All of the content (graphics) for an application can be contained within the volume of a prism. The universe can automatically mask out graphics that extend outside of the prism. Because applications have no knowledge of other applications in the world, the universe can manage the interactions that occur between different prisms in different applications.

[0134] A user interface design for placing a prism may require the prism to swing in a physical manner (like an object on a string) while it is being moved in its placed state. Instead of trying to predict the kind of physical behavior different applications will want, the prism can feed movement information to the application (through a binder interface) while it is being placed. The application can then behave appropriately.

[0135] There may also be physical behavior between the prisms as they are being placed, which may invalidate the application's physics implementation and cause the application to stop receiving movement data.

[0136] The prisms may initially resist crossing. If the user continues to press two prisms into the same place, the prism may snap into place with the prism it crosses. This may be done in a resilient manner (e.g., similar to soap bubbles interacting with each other) and loosely based on physics.

[0137] Audio emitters can be placed in the application's scene graph as child nodes. These nodes can be local to the root node transform. Thus, prisms can be moved, and moving the prism does not require the application to update the transform of the audio node. The universe can be responsible for the final transformation from the audio emitter to world space. Prisms can also be responsible for constraining the audio node to its boundary. Applications may not emit audio from points outside their respective prisms.

[0138] In some embodiments, it may not be desirable to spatialize the audio. For example, if a user places a virtual TV on a wall and is centered on the TV image, the TV audio may be presented to the user without modification. This is likely to provide a better audio experience for the user. In the case of ambient sounds, the audio signal already contains spatial information. The sound may be emitted from virtual speakers placed in optimal locations relative to the TV.

[0139] In some embodiments, when a user presses a button to control audio intensity, the universe may check the head pose, determine which prism the user is looking at, and send a volume up or volume down event to the corresponding prism. The prism may forward that information to the application launched in the prism, and the application may decide how to interpret it. If no application is in focus in the landscape, the volume button setting may adjust the global volume.

[0140] In some embodiments, one difference between a traditional 2D window and a prism 113 is that in a 2D window, the border defining the boundary of the 2D window is intended to be visible by the user, providing a clear boundary to contain content within the 2D window and separate it from content outside the 2D window's border. However, in some embodiments, the border of a 3D window (e.g., a prism 113) is intended to be invisible. If the outlines (e.g., borders) of all prisms were visible to the user, it would destroy the illusion of "reality," and virtual content displayed within a prism whose borders are visible would appear to be computing / digital / virtual content instead of being real. In some embodiments, the borders may be displayed, for example, to enable user interaction, as needed.

[0141] Another difference is that 2D windows are generally intended to be controlled and / or interacted with by the user. For example, a close button may always appear in the upper right corner of a traditional 2D window, or a menu bar may be displayed on the top border of the 2D window. However, in Prism, the user generally does not interact with the prism and its borders. Instead, a secondary menu (e.g., an app options menu) may be temporarily pulled down for the user to control and manage / operate the prism from a list of options.

[0142] Furthermore, a 2D window is independent of its surroundings. For example, what is displayed on a computer screen does not automatically change if the user moves the screen. However, prisms must be placed in relation to the real world. For example, each prism may be placed in relation to (1) an object in the real environment, such as a wall or table; (2) a virtual object created to provide a background or canvas for the prism to anchor; and / or (3) the real world relative to the user. In some embodiments, prisms may be placed in relation to the passable world and the real world.

[0143] Still further, in some embodiments, prisms may not be allowed to overlap / interpenetrate each other, with the exception that prisms may only overlap slightly for physics purposes. For example, in some embodiments, when virtual content within two or more prisms collide, the virtual content may appear to bounce between the two virtual contents as they appear to collide with each other. Here, the prisms may only overlap slightly to create the effect of bouncing between the two virtual contents. In some embodiments, when the bounding boxes for two or more prisms collide, the prisms, and therefore the prisms' contents, may appear to bounce. However, 2D windows on a computer may overlap, and often 2D windows may cascade on top of each other, hiding each other from the user's view. In some embodiments, when two prisms are anchored in the same location in the user's landscape 110, one of the prisms may be displayed, while the other prism is minimized from view, and an icon or text or image (or any other visual indicator) is displayed to indicate to the user that another prism is anchored in the exact same location. In some embodiments, an infinity prism may be implemented to render additional graphics around the prism, for example, when the prisms collide. In some embodiments, an infinity prism may be a prism whose boundaries are set to infinity. For example, if two prisms are close enough to collide, the universe may render a glow within the region of space between the two prisms. To handle these exceptions, the universe may create an infinity prism that may encompass all space around / encompassing the two prisms, the user's entire field of view (what the user can currently see), the user's entire kinesthetic field of view (what the user can see as they move around), etc. This may allow the universe to pull graphics anywhere between the two prisms. In some embodiments, the infinity prisms may not collide or interact in any way. In some embodiments, the infinity prisms do not have secondary UI, etc.In some embodiments, only the universe may have access to the infinity prism. The infinity prism may be created at universe initialization time and may exist at all times until the universe is shut down. In a second example, the infinity prism may be useful for moving characters (e.g., avatars, personal assistants, butterflies, animals, etc.) between other landscape apps, for example, to explain to the user the content of each application and / or how to use the application.

[0144] FIG. 4 shows an exemplary launcher menu as seen, for example, by a user wearing head-mounted system 160 for launching applications and displaying virtual content according to some embodiments. The launcher menu may exist as its own application, with elevated privileges requiring the launcher menu to launch other applications. The launcher menu 400 may include a status bar 410 and may have body-centered bounding volumes (e.g., prisms) called panels. In some embodiments, a launcher panel may include several different applications that give the appearance of a single application. For example, each panel may be a separate application but give the appearance of a single launcher application. The universe may make an explicit create call for each panel during the launcher menu invocation (e.g., a request to open the launcher menu).

[0145] In some embodiments, a separate application process may render into each panel. The process may use the same content rendering techniques as the rest of the universe, with one or more of the following exceptions in some embodiments: (1) the launcher menu may trigger the launch of other applications; (2) launcher menu icons and widget icons may be extracted and placed in the landscape (placed icons may be referred to as shortcuts); (3) the launcher menu itself may be placed and moved in the landscape; (4) when content extraction begins on a launcher icon, the launcher may load placeholder content that is anchored to the user's hand, six-degree-of-freedom controller, or other controller, so that when extraction is finished, the universe launches the application into the prism, which replaces the placeholder content; and (5) the user may swipe between panels in the launcher menu to expand individual panels. In some embodiments, the panels may not be implemented as a carousel. Alternatively, the launcher panels may be co-located such that the primary panel is always fixed as the center panel and content from minimized panels is animated in and out of the center panel as the user swipes left, right, up, or down. In some embodiments, the launcher application itself may be located by the user within the user's passable world, thereby creating a launcher shortcut or a pinned launcher within the user's passable world and / or landscape.

[0146] In some embodiments, the launcher menu 400 may organize a series of body-centered panels as a panel carousel. The panel carousel may display one panel at a time, with each panel launching its own application process and / or each panel being a separate application. In some embodiments, only one current panel 405 may be expanded and visible at a time. Two body-centered placeholder panels (e.g., 440a and 440b) may be located a fixed distance from the current expanded panel 405, on either side of the current expanded panel 405. The placeholder panels 440a and 440b may be individual prisms created by the universe that represent two panels in abbreviated form adjacent to the current expanded panel 405. In some embodiments, when the user swipes to the next panel, the panel represented by the placeholder may become the current panel. The physical model of the placeholder panel may remain the same regardless of which panel is represented. In some embodiments, the only visual difference between the placeholders representing different panels is the label (e.g., Recent and Social, as shown in FIG. 4). Other embodiments may have different panels in the same location but with different icons specific to the panels (see FIG. 11).

[0147] In some embodiments, the launcher menu 400 manages the flow of panels, ideally caching left and right panels. The panel carousel may include a recently accessed applications panel 420, a general applications panel 425, a social applications panel 430, and a store panel (not shown). The recently accessed applications panel 420 may include icons of applications recently accessed by the user. The recently accessed applications panel 420 may be shown in a minimized state when the current panel 405 for the general applications panel 425 is currently displayed. The recently accessed applications panel 420 may expand to show its icons when the user swipes in the corresponding direction (e.g., right) to bring the recently accessed applications panel 420 front and center, or according to other embodiments, the content for each panel may animate in and out of the center panel. In some embodiments, the recently accessed applications panel 420 may be displayed first, and the applications panel 420 may include a ring of applications. Any number of panels may be used, and the panels may be displayed in any order. The order and number of panels may be based on user testing, user ease, and / or user preference.

[0148] In some embodiments, the social application panel 430 may be shown in a minimized state when the current panel 405 for the general application 425 (or any other panel) is currently displayed. The social application panel 430 may expand to show its icons when the user swipes in the corresponding direction (e.g., left) and brings the social application panel 430 front and center, or in some embodiments, content associated with the social application is animated / displayed in the center panel while labels on the placeholder panel are changed accordingly.

[0149] In some embodiments, icon 450 of general application panel 425 may represent an icon within an icon grid application similar to icon grid application 260 from FIG. 2. Icon 450 may represent an application installed on the mixed reality system. A user may launch an application by selecting a particular icon 450. In some embodiments, the user may extract icon 450 from the panel and place icon 450 in the user's landscape, at which point universe may begin the launch process for the application corresponding to extracted icon 450. Once the corresponding application has completed its rendering of its content, extracted icon 450 may be replaced with a prism, and the application's rendered content is displayed within the prism.

[0150] As discussed above, in some embodiments, the status bar 410 may comprise status indicators for the mixed reality system. The status indicators may be populated, for example, by first querying a first service for operating and maintaining WiFi services, a second service for maintaining Bluetooth services, and a third service for status. As these services notify the status bar of updated status, the status bar may adjust accordingly. The status bar may include status indicators for services such as audio volume, current time, WiFi connection / strength, Bluetooth, and power. Other services may be included in the status indicators in combination with or instead of the above-mentioned services.

[0151] In some embodiments, based on a request to launch an application from the launcher menu 400, the icon grid application may communicate with the universe, for example, through the prism service and / or the universe service, to launch the application. The universe may create a prism using a specified package name and / or component name. The universe may provide the prism to the application using the application's listeners. The prism may now be in "installed mode." Once the user installs the prism, the user may modify the prism in any number of ways, for example, using a secondary UI application options menu.

[0152] 5A-5B illustrate exemplary panel carousel changes according to some embodiments. FIG. 5A illustrates an exemplary view of a launcher menu with placeholder panels represented as left 520 (e.g., 440a from FIG. 4) and right 530 (e.g., 440b from FIG. 4) panels before a left-to-right swipe. Current active panel 505 illustrates an exemplary icon grid depicting an icon in the icon grid as "BB" in active panel 505. In some embodiments, the icon grid can be one or more icons forming a grid, a list, one or more columns with one or more rows, a circle of icons, and / or any other shape. Minor panel 535 illustrates available panels in the panel carousel (in some embodiments, minor panel 535 is not shown to the user). The panel position indicator 540 shows the center panel represented as "C" pointing to the "B" panel, the left panel represented as "L" pointing to the "A" panel, and the right panel represented as "R" pointing to the "C" panel, with the "D" panel not currently in view and therefore not specified in the panel position indicator 540 (and in some embodiments not shown to the user).

[0153] FIG. 5B shows an example view of the launcher menu as shown in FIG. 5A after a left-to-right swipe. The new active panel 555 is now the “A” panel, with “AA” as the icon in the icon grid of the new active panel 555. Two placeholder panels, the left 570 panel, shows the “D” panel, and the right 580 panel shows the “B” panel from the previous active panel 505 in FIG. 5A. In addition, a small panel 585 now shows the available panels in the panel carousel, as panels A, B, C, and D. However, the panel position indicators 590 have changed accordingly, indicating that the center panel, represented as “C,” now points to the “A” panel, the left panel, represented as “L,” now points to the “D” panel, and the right panel, represented as “R,” now points to the “B” panel.

[0154] In some embodiments, the placeholder panels themselves may remain constant, but only the labels reflect the underlying panels, which change. For example, in FIG. 4, if the user performs a left-to-right swipe, such as the left-to-right swipe in FIG. 5B, the new active panel becomes the "Recent" panel, the right placeholder panel is displayed with a new panel title of "General," and the left placeholder panel is displayed with a new panel title of "Store" (assuming only the panel carousel has the four panels disclosed in FIG. 4). The images of the placeholder panels may remain the same, but the titles of the left and right placeholder panels may change accordingly. Panel shifting may be based on a user motion switching action, e.g., a left-to-right swipe. Those skilled in the art will appreciate that any other method of switching between panels may also be used.

[0155] 6 shows a flowchart for an approach for starting a mixed reality system after a previous use thereof, according to some embodiments. Start-up process 600 shows a process for powering on / starting a mixed reality system some time after a user previously used the mixed reality system and powered it off. At 610, the mixed reality system identifies the user's location, establishes a head pose, and determines the location and / or orientation of the user (e.g., the user's head). Head-mounted system 160 may include a GPS for at least partially determining the user's location.

[0156] At 620, the universe may access the passable world system (e.g., external database 150) and retrieve instances of prisms previously displayed in the current user's landscape. The passable world may return the instance data, so the universe may reconstruct the previous state of the prisms in the user's landscape by reconstructing the user's local database 137 with the retrieved prism instance data. The passable world system persistently stores information about the user's environment, such as which prisms were previously active in the user's landscape 110, as well as the state of each prism when the user previously shut down the system. The user's prism instance data stored in the local database 137 may be erased when the user shuts down the mixed reality system, so that when the mixed reality system restarts, the user's prism instance data in the local database 137 may be reconstructed based on the instance data retrieved from the passable world system (e.g., external database 150 from FIG. 1 ).

[0157] In some embodiments, the instance data may include, for each prism, one or more of the following: (1) prism attributes (the prism attributes may comprise location, orientation, range width, range height, range depth, anchoring type, and / or anchoring position); and (2) application-specific attributes comprising state information of virtual content previously rendered by the respective application. For example, if the application rendering into the prism is a video streaming application, the application-specific information may include the title of the video, the point in time at which the video was played, and the aspect ratio of the video being rendered. The application-specific information may be a list of key / value pairs that store state information for each application. The list of key / value pairs may be specific to each application, and the key / value pairs may provide state information for the application's content being displayed or rendered within the prism.

[0158] In some embodiments, each user 103 may have their own prism instance data stored in their local database 137, while master prism database instance data is stored in the passable world system (e.g., external database 150) and includes other prism instance data for other users of the mixed reality system and / or prism instance data for other locations in addition to the user's current location. Prism data for users of the mixed reality system may be stored in the passable world system. In some embodiments, a first user may want to share virtual content displayed in a prism from the first user's landscape with a second user who is using the mixed reality system at the same location (or, in some embodiments, a different physical location than the first user). Because the prism information is available to the first user in the first user's local database, sharing the prism with the second user may upload the instance data for the prism from the first user to the passable world system. The instance data for the prism may then be transmitted or read by the second user and stored in the second user's local database, and the second user's universe application may reconstruct the prism's instance data into the second user's local database, thereby displaying the prism in the second user's landscape.

[0159] At 630, the prisms may be restored at the user's current location. Once the instance data for prisms previously deployed at the user's location has been reconstructed in the user's local database 137, each application corresponding to the prisms previously deployed at the user's current location may be launched. Since the mixed reality system simply starts up, applications that previously provided content to the mixed reality system may need to be started, but at this point, only applications that previously provided content to the user's landscape need to be launched. As part of launching an application, the application may request that the universe create prisms for the application to render its content; the prisms to be created correspond to prisms previously deployed prior to the user shutting down the mixed reality system for display in the user's landscape. Once the application is launched and the prisms are created, the application may provide content to the universe for rendering into the respective prisms.

[0160] In some embodiments, an application may provide content to more than one Prism deployed at a user's location, in which case the application may deploy multiple instances or processes of the application and render content into one or more Prisms for each instance or process of the application.

[0161] At 640, the prisms may be displayed in the user's landscape in the locations where the prisms were displayed prior to the user shutting down the mixed reality system. Because the prisms and applications rendering into the prisms persist, the mixed reality system may reconstruct the user's landscape by deploying / displaying the prisms, along with their respective virtual content, in locations of the user's landscape based, at least in part, on the respective prism attributes of each of the prisms. In some embodiments, a placeholder prism, such as loader volume 210 from FIG. 2, may be displayed in the location of the currently loading prism at startup. In such embodiments, once a new prism is created and restored, displaying the new prism in place of the placeholder prism may be a simple matter of replacing the respective placeholder prism with the newly created prism in the place of the placeholder prism. If the prism has attributes marking it as anchored by the user in place of the landscape, the prism may be restored to a location relative to the user's current location.

[0162] In some embodiments, applications rendering within prisms may be deployed in a state consistent with the state they were in when the user previously shut down the mixed reality system. For example, a user's view of the user's landscape may have four prisms deployed, with one of the four prisms having a sleeping application while the other three prisms each had an actively rendering application. When the mixed reality system resumes and the four prisms are displayed, three of the prisms may have an actively rendering state, while the fourth prism may have an application that is sleeping until the user reactivates the fourth prism's sleeping state, or the prisms may be configured such that their state may automatically change to an actively rendering state if the user looks into their general vicinity. Changing the state of a prism is further disclosed below. In some embodiments, other methods may be used to change the state of a prism.

[0163] At 650, the user's local database may be updated with prism information continuously (or at regular intervals or when triggered by an event) and automatically as the user interacts with the prisms in the user's landscape. For example, when an application for a prism changes state from actively rendering to active but non-rendering or sleeping, the user's local database is updated with updated state information about the application in that prism. Updated information from the user's local database may immediately or periodically update the passable world system (e.g., external database 150) to ensure that the passable world system is kept up to date with changing states of applications and prisms in the user's landscape. Those skilled in the art will understand that this is merely one example of how prism attributes or application-specific attributes may be changed and updates made to the local database and then to the external database; many other similar types of changes to prisms or applications that provide content to prisms may be made.

[0164] The problem of keeping virtual content from one application displayed in a 3D spatial environment private from virtual content from a different application displayed in the 3D spatial environment, while still being able to interact with it, can be solved by having one program manage content creation (e.g., application 140 from FIG. 1) and a separate program manage content display (e.g., universe application 130 from FIG. 1).

[0165] FIG. 7 shows a flowchart for an approach for displaying virtual content provided by an application in a mixed reality 3D spatial environment by a universe application according to some embodiments. At step 710, a user's universe application may receive a request to display virtual content within the user's landscape (e.g., a 3D spatial environment). In some embodiments, the request may be initiated by a user interaction with a launcher menu, where the user selects an icon from an icon grid to launch an application and provide the virtual content within the user's landscape. As discussed above, one application (e.g., application 140 from FIG. 1) may manage content creation, and a separate application (e.g., universe application 130 from FIG. 1) may manage content display. This separation of processing may allow multiple applications to create content for display within the user's landscape, while management of the display of the content is managed by a separate application, ensuring that multiple virtual contents displayed within the user's landscape appear accurate and realistic to the user, especially when the virtual contents begin to interact with each other.

[0166] At 720, the universe may create a prism so that content created by the application can be displayed within the prism's boundaries in the user's landscape. It is important to note that while a prism may include boundaries that define the boundaries of the prism's three-dimensional space, the boundary lines of the prism's boundaries may not be visible to the user. This may be important because virtual content rendered inside a prism to be displayed within the user's landscape should appear as real as possible to a user viewing the virtual content within the user's landscape. If the boundaries of the bounding three-dimensional space that bounds the virtual content were visible to the user, the virtual content would inevitably appear computer-generated within a confined space. Thus, a prism may include boundaries and boundaries that confine the virtual content within the prism's boundaries, but the boundaries may not be visible to the user. The prism's boundaries may be visible in certain situations, such as during system development and testing. In some embodiments, the prism's boundaries / boundaries may be displayed to the user, for example, during installation, so the user can more easily understand where the prism will be located.

[0167] In some embodiments, a single application may be requested by a user to display multiple pieces of virtual content within the user's landscape. Each piece of virtual content displayed by a single application may be associated with and displayed by its own individual prism, or may share a single prism. Thus, in some embodiments, if an application requests additional virtual content to be displayed, the universe may create one or more additional prisms for the application, and each piece of additional content may require its own prism to be deployed / anchored within the user's landscape.

[0168] The created prism may be provided with a unique identifier of the application requesting to display the content, thereby enabling the universe to uniquely identify the prism for rendering the content received from the application when the application provides the content for display within the user's landscape. Furthermore, prisms may be automatically created to have a predetermined set of functionality, which may be predetermined by the application and / or the universe in some embodiments. For example, the set of functionality may include a default minimum / maximum size allowed for the prism, an aspect ratio for sizing the prism based on the distance the prism can be anchored within the user's landscape relative to the user, an anchoring type for anchoring the prism within the user's landscape, etc. The set of functionality may be stored in a local database as prism attributes. Furthermore, depending on the application invoking the creation of the prism, certain application-specific attributes may also be created for the prism.

[0169] At 730, the universe may receive the virtual content created by the application. Upon receiving the content, the universe may render the virtual content to be displayed within the boundaries of the prism. As discussed above in FIG. 2, in some embodiments, the application may create content and send it to the universe via a centralized rendering service client, which launches within the universe. From that point on, the universe may render the content into the prism created for the application requesting to display it. The universe may also be responsible for ensuring that the prism is displayed correctly within the user's landscape. For example, the universe may ensure that a prism does not overlap with other prisms in the user's landscape. The universe ensures that a new prism cannot be placed in an overlapping position with respect to existing prisms or real-world content unless an exception (e.g., a common anchor point) applies, as described above. The universe may not allow a user to place a prism where the prisms overlap. In some embodiments, client-server communication with the application may not be required. If the operating system (e.g., the part of the system that manages rendering) is configured for multithreading, applications can render themselves directly into the prism without a client / server structure. A central authority such as a universe server can then be involved in managing collisions, movement, input, etc.

[0170] At 740, based on either user input or default anchoring rules that may be applied to a particular application or prism, the prism may be associated with (1) a fixed location in space within the user's landscape, (2) a physical object within the user's landscape, (3) a virtual object displayed within the user's landscape, or (4) a fixed distance relative to a particular object, such as the user's body or body part (hereinafter also referred to as body-centered anchoring). In some embodiments, body-centered anchoring may be relative to a body part of the user (e.g., body fixation) or relative to head / view position. The prism may be associated with a body-fixed position such that the prism is fixed relative to the user's body. When the user moves their head, the content may not move, but when the user walks, the prism may move with the user and maintain a fixed position relative to the user's body. As another example, the prism may be associated with a head-fixed position such that the prism is fixed relative to the user's head or posture. When the user rotates their head, the prism may move relative to the user's head movement. When the user walks, the content may also move relative to the user's head.

[0171] At 750, the prism is placed / anchored in the user's landscape. Depending on how the user launched the application, the prism may already be associated with a particular anchor / location. For example, the user may launch the application by dragging the application icon from an icon grid in a launcher menu and place the icon on a particular location in the user's landscape, at which point a prism may be created at that location and content may be rendered in the prism at that particular location.

[0172] FIG. 8 illustrates an example diagram for managing application states according to some embodiments. The problem of high computational load for managing, rendering, and displaying augmented reality content can be solved by automatically changing the state of a prism based on the user and, in some embodiments, the relative position and location of the prism to the user's field of view. Diagram 800 shows a bird's-eye view of a user's location according to some embodiments. User 805 is shown as residing within a volume grid that is a rough representation of a planet (e.g., a physical environment) in the x, y, and z axes. Grid lines are shown for illustrative purposes to indicate that the distance from user 805 to the prism has 870 different prism states. Here, a 2D representation of the user's current location is depicted. Those skilled in the art will understand that this diagram could be a 3D representation of the user's current location. However, for simplicity's sake, FIG. 8 is illustrated to show 2D space.

[0173] User 805 is shown residing at the center of a 3x3 cell exterior zone 810. In addition, exterior zone 810 is divided into two circular areas, including an active zone 820 and a buffer zone 830. Exterior zone 810 comprises a grid of 3x3 cells, with user 805 located at the center of the grid. As user 805 moves through the user's landscape, other prisms associated with each application may enter and exit various zones, as depicted in FIG. 8. Active zone 820 illustrates the area (e.g., a circular area) surrounding user 805 and indicates certain prism states 870 for prisms located within active zone 820. For example, sleeping prisms within user 805's active zone 820 must be woken up, while other awake prisms must remain awake even if their state is active but not rendering within active zone 820 870b.

[0174] The buffer zone 830 is a zone further away from the user than the active zone 820. Prisms within this zone may not require a state change. For example, activated prisms remain activated, e.g., one prism located within the buffer zone has an active but non-rendering prism state 870b. As another example, sleeping prisms may remain asleep, as in two prisms within the buffer zone 830 with a sleeping prism state 870a. Prisms outside the buffer zone 830 but within the external zone 810 may remain asleep because they are far enough away from the user 805 that they may not need to change state. Finally, prisms located within the outer associated cell 815 may have a non-loading, sleeping, or background prism state because they are located far enough away from the user 805 that it may be beneficial for the mixed reality system to conserve processing resources by not activating those particular prisms until the user 805 is within a closer distance of the respective prism.

[0175] Prism state legend 860 discloses the various different prism states of the prism based on color scheme, with a red prism indicating a sleeping prism state 870a, an orange prism indicating an active but non-rendering prism state 870b, a green prism indicating an active and rendering prism state 870c, a purple prism indicating an unloaded prism state 870d, and a blue prism indicating an active and non-rendering background prism state 870e.

[0176] User 805 may have a particular field of view 840, and prisms within field of view 840 may lie within frustum 850. A frustum is a 3D region visible on the user's 805 screen (e.g., via head-mounted system 160 from FIG. 1). Activation of a prism within frustum 850 should be rendering, as indicated by a prism with prism state 870c that is active and rendering. Since there is one prism with prism state 870a that is located within frustum 850 and within buffer zone 830, that particular prism may wake up and render when it is located within both buffer zone 830 and frustum 850. In some embodiments, the head pose frustum may be used to cull the rendering of a prism, and frustum culling is a method of determining hidden surfaces, such as the hidden surfaces of a prism.

[0177] Table 1 below illustrates a prism state change chart, where the rows in the first column describe the current prism state. The rows in subsequent columns describe the change in prism state from the current state to a new state as the prism moves from the current zone into a new zone, as indicated by the subsequent columns. The values in the rows in subsequent columns describe the new states to which the prism may change (if at all) based on the relative distance from the prism to the user 805. [Table 1]

[0178] For example, if a prism is in a current state of "sleep" and based on the user's 805 movement, which may place the prism within the active zone 820 for the user 805, the prism state of the prism may be changed to an "active, but non-rendering" state because when a sleeping prism is within the user's 805 active zone 820, it should wake up. Additionally, following the same example, if the prism is in a current state of "sleep" and the prism comes within the buffer zone 830 and frustum 850, the state remains unchanged because, within the buffer zone 830, a sleeping app remains asleep even though it is within the buffer zone 830 and frustum 850. However, once the sleeping prism transitions into the active zone 820 and frustum 850, the prism may begin to wake up and the prism state may change to an active, rendering state 870c.

[0179] FIG. 9 shows a flowchart for managing prism states relative to a user location according to some embodiments. At 910, the spatial location of a prism may be recorded in a volumetric grid. The spatial location may be the anchoring location of a particular prism placed / anchored within the user's landscape. For example, a prism may be anchored to an object within the user's landscape, such as a wall or table. As another example, a prism may be anchored to a fixed location in space that is not associated with any particular object within the user's landscape, such as a floating 3D sphere of planet Earth. When a prism is initiated, its spatial location is recorded in a volumetric grid, which may have x, y, and z axes relative to the world to identify the prism's location within which the prism's content may be rendered. Each prism placed within the user's landscape may have a specific location stored (e.g., as a volumetric grid) within a prism data structure stored in the passable world system. The volumetric grid may represent a rough representation of the planet in x, y, and z axis format, as discussed above. Any other reasonable coordinate or relative placement system may also be used.

[0180] At 920, cells of a volumetric grid representing the 3D spatial area are identified. A cell may include one or more prisms if the prisms are spatially close to one another. In some embodiments, the width of a cell may be equal to or greater than the radius of the active zone. In some embodiments, the width of a cell may be less than the radius of the active zone. The radius of the active zone indicates a measurement of distance from a user using the mixed reality device to a preconfigured distance (e.g., 25 meters), which would define a circular and / or spherical (or other suitably shaped) area of space as the active zone 820 as depicted in FIG. 8. Alternatively, the volumetric grid may be divided into cells having predetermined lengths to further establish boundaries and perimeters corresponding to distances from a user using the mixed reality device and establish different zones for implementing embodiments of the present disclosure.

[0181] At 930, as a user using a mixed reality device moves to a location cell, the distance of each prism's known location in that cell and neighboring cells is determined. As a user using a mixed reality device moves through the user's landscape, a distance check is performed on the last known location of each prism to manage the status of prisms in the user's neighborhood. To prevent performing distance checks across prisms worldwide, the system may minimize the amount of distance checks / calculations / determinations by using a volumetric grid and limiting searches / checks / calculations / determinations to only the cells surrounding the mixed reality device (e.g., the cell and neighboring cells). In some embodiments, only prisms in the cell occupied by the user are distance checked. In some embodiments, only prisms in the cell occupied by the user or cells adjacent to the cell occupied by the user are distance checked. In some embodiments, prisms in all of the cells in the user's FOV / view frustum are distance checked. In some embodiments, prisms within a certain distance of the user, such as within 10 feet or 25 feet of the user, may be distance checked.

[0182] At 940, the state of the prism may be modified depending on the distance between the prism and the user. In addition to distance, other factors, such as the current state of the prism relative to different zones and the user's field of view, may also determine whether the state of the prism may be modified. For example, if the current state of the prism is sleep and the user moves to a location where the prism is located within the active zone 820 and frustum 850, the state of the prism may be changed from the sleep state to the active and rendering state according to Table 1. However, if the sleeping prism is located within the buffer zone 830 and frustum 850, the state of the prism may not be changed because the current state of sleep remains the sleep state when the prism is moved into the buffer zone and into the frustum zone according to Table 1. The buffer zone, as disclosed above, primarily prevents intermittent or rapid changes to the state of the prism. The user's head pose may modify the state of the prism because the user's head pose helps define the user's field of view, which ultimately helps define the frustum zone 850. As disclosed above, a user's head pose is a measurement of the location and / or orientation of the user's head. Head pose information can be used to render a scene to match the user's dynamically changing head location and orientation, providing increased immersion within a virtual / augmented / composite space.

[0183] In some embodiments, a wall may separate the user from the prism, such that the prism remains in a sleep state even if the user's distance places the prism within the active zone 820 and / or the user's viewing direction places the prism within the frustum zone 850; the prism's state may not be changed from a sleep state due to the obstruction of a wall that prevents the user from seeing the prism. Further computational processing may be avoided by the system recognizing that other factors, such as a wall or object blocking the prism, may keep the prism's state unchanged even though distance and head pose may indicate a state change for the prism. (Manage Prism Relationships and Groupings)

[0184] In some embodiments, a centralized rendering system may not have support for transforming / managing client prisms 215 as part of a larger (e.g., universe space) scene graph. A universe prism may be an instance of a client prism with additional logic and functionality. In a universe, the ability to transform a group of client prisms 215 may be handled via a transformation tree. For example, each panel of a launcher may be a separate application, with each panel corresponding to its own prism. It may be beneficial for a universe to be able to group multiple prisms associated with a launcher into groups so that they can be managed together, whereby when one prism in the group is modified, other prisms in the group may be affected and therefore may need to be modified as well. This may enable multiple applications to look and behave as if they were a single application. For example, if a launcher is anchored to the user's body (e.g., body motion), as the user moves throughout their environment, not only the location of the launcher but also the locations of its panels corresponding to the multiple prisms may need to be modified / transformed as a group.

[0185] As another example, an application that requires a prism may have a secondary UI (with a close button, pin / unpin, settings, etc.) located relatively close to the application's prism. A universe may group these two prisms together as a group and manage the two prisms as a group with two nodes. Managing the association of multiple prisms in a hierarchical / relational manner and / or logical grouping structure may be handled using a transformation tree and / or a group tree (discussed below).

[0186] In some embodiments, a transform tree is a tree data structure configured to transform / organize prisms and / or groups of prisms in a universe and may be rendered by a central rendering system. The transform tree structure may have a scene graph of nodes, with each node providing a movable local space corresponding to a prism. A node in the scene graph may correspond to a prism. A prism may be a tree node. There may be other tree nodes that are not prisms. These "other tree nodes" may be used to group content or as intermediate transformations. Additionally, nodes may be grouped with other nodes so that when one node is transformed, other nodes in the same group are transformed along with it. In some embodiments, a group tree may manage the grouping of nodes.

[0187] A group tree can be a tree data structure that stores logical groupings of nodes.

[0188] In some embodiments, prisms may be treated as independent content displayed within the user's landscape without overlapping, which may introduce the problem of virtual content from one application not being able to tightly interact with virtual content from different applications. However, creating multiple smaller prisms that fit tightly around virtual content from a single application may solve this problem. For example, there may be multiple prisms open for a single application, but the user may perceive the multiple prisms as one unified system (e.g., parent / child prisms in a transformation tree). Additionally, prisms may be grouped together such that some prisms may essentially be peers of each other. If all of the content from one application is within a large box, there may be more "air" or dead space within the prism. This may prevent content from different applications from interacting with content within those areas and therefore prevent certain types of "realistic" interactions from occurring.

[0189] For example, an interaction can be a simple collision of two 3D cars. The collision may not appear "real" to the user if the prisms through which the two 3D cars are rendered each have such a large "dead space" relative to the 3D cars that the user may not be able to see the two cars actually touching each other between the invisible boundaries of the prisms. However, if the prisms are smaller and fit tightly around each of the 3D cars, from certain viewing angles, it may appear to the user that the two 3D cars slightly touch each other and bounce off each other as a result of the collision.

[0190] 10 shows an example tree node of a scene graph according to some embodiments. Tree node 1000 depicts an example tree node for a launcher menu. Universe node 1002 is the root node in this tree node of the scene graph of a user's mixed reality system. When the mixed reality system starts, a launcher application may be started to display as a launcher menu to the user. As the launcher application is launched, a launcher tree node 1004 may be created that represents the launcher menu's location in the scene graph. Launcher tree node 1004 may serve as an anchorage for other nodes that are subordinate to or children of the launcher menu.

[0191] For example, an icon grid application may also be launched. An icon grid tree node 1009a may be added to the tree node along with the scene graph identifier (SGID) of the launcher application. Because the icon grid application may be displayed as one panel of a carousel of panels for the launcher menu, the icon grid tree node 1009 may be a child node of the launcher tree node 1004, as depicted within child node 1008, shown with a dashed border, labeled as a launcher pane. More of the launcher's subordinate applications may be launched and added to the tree node as child nodes of the launcher tree node 1004 (e.g., recently accessed applications tree node 1027 and social applications tree node 1029). Groups of nodes associated with a launcher menu may be grouped together as a launcher group 1006.

[0192] When a launcher menu is moved (e.g., billboarded), the nodes in the launcher group 1006 may be moved as a group. Setting a launcher node as a fixed node may be necessary to provide a precise means for precisely positioning / billboarding a launcher menu, rather than applying logic to one of the launcher panels and hoping for the best to maintain the grouping of other related panels. Child nodes (e.g., icon grid, recently accessed applications, and social applications) are automatically repositioned relative to the launcher tree node. The universe may own the entire scene graph of multiple prisms. In some embodiments, each prism may start with a null parent and zero children, effectively resulting in a scene graph with one node. In some embodiments, when an application requests a prism, the universe may create (1) a node for the requested prism and (2) a node for the secondary UI. The universe may then position the secondary UI relative to the prism. In some embodiments, the universe may place the secondary UI node as a child of the prism node. (Transformation tree)

[0193] As discussed above, a transformation tree is a tree data structure configured to transform / organize prisms and / or groups of prisms in a universe. The transformation tree structure can be thought of as a scene graph with nodes, where each node provides a movable local space that may correspond to a prism. A node in the scene graph may correspond to a prism, or it may be a node used only to transform a group of prisms or other tree nodes. In this case, a node may be grouped with other nodes under a parent node so that when one node is transformed, other nodes in the same group are transformed along with it. In some embodiments, a prism is in a transformation tree with only one prism (i.e., itself), but there is the ability to parent one prism with another prism when the need arises. In some embodiments, a prism scene graph / transform tree may differ from a traditional scene graph in that when a child prism changes its parent, the child's world transformation may not change. For example, if a prism renders in front of the user, the prism will not move from its current position when its parent changes. In some embodiments, the matrix of each prism in the transform tree may be column-major, meaning that the calculation of the world matrix of a prism in the tree must be done by adding the parent transform to the left of the prism's local transform. For example, the world transform of a prism may be equal to the parent's world transform times the prism's local transform.

[0194] In some embodiments, the basic tree logic may be defined as one or more of the following: (1) a node may set itself as a child to another node; (2) a node may set a different node as a child to itself; (3) a node may acquire its parent; (4) a node may acquire its child; (5) a node may remove any of its children (this does not make the children orphans: they may simply attach to this node's parent or any other node in the tree, including the root universe node, which may place it in world space); and (6) a node may remove its parent. Insertion and removal of nodes may update the local transformation of the node and its children. This means that changing the parent of a node does not change its world transformation, and in fact the world transformation may be preserved. In some embodiments, any action that changes the local transformation of a node may update the world transformation of the node and its children.

[0195] In some embodiments, each node in the transform tree may include one or more of the following matrix components: (1) local transform; (2) rotation matrix (which may be used to calculate the local transform); (3) scale vector (which may be used to calculate the local transform); (4) position vector (which may be used to calculate the local transform); (5) local-to-world transform (which may be cached for convenience); and (6) local-to-world scaling (which may be cached for convenience). In some embodiments, the local transform may be considered the transform applied to its parent node. If the node has no parent, the local transform may be equal to the local-to-world transform. The local transform may be calculated as follows: Local transformation = position x rotation matrix x scale

[0196] This formula may be read as "Using the parent node as its origin, transform this node by first scaling it, then rotating it, and then changing its position relative to the origin node." Rotation, scaling, and positioning may be kept separate to make it easy to provide getters / setters to modify any of them separately through an interface. As a node changes position, the child node world transform may be updated accordingly. When these matrices are multiplied in order, a local-to-world matrix may then be constructed. For efficiency purposes, the calculated local-to-world matrix may also be stored. In some embodiments, the following equation may be used to calculate a node's local-to-world transform: World conversion = Parent world conversion x Local conversion

[0197] Note the order of matrix multiplication. In some embodiments, the matrices are column-major, implying that x, y, z, and position are laid out in each column. This means that parent node matrices always appear to the left of their child matrices. Figure 10B illustrates the small transform tree 1010 and how each world transform for each node is calculated.

[0198] Other suitable methods or systems that utilize a transformation tree may be used. (Group Tree)

[0199] In some embodiments, an additional feature of transform trees is the ability to group any prism with any other prism within a transform tree. Prisms within a group can be in the same or different transform trees. Additionally, a prism can be a child or descendant of another prism within the same group and transform tree. In some embodiments, when a transform is applied to one of the prisms in a group, all of the other prisms in the group are transformed along with it. Prism group logic also handles updating transforms in any transform tree to which a group member belongs. Any prism that is not in a group but is a child of a prism that is in the group may also transform along with the group. When a prism in a group is transformed, its children are also transformed. For example, if prisms A and B are in a group, and prism C is a child of prism B but not in the group, and prism B is moved one meter to the left, both prisms B and C are moved one meter to the left. However, if prism C is moved one meter to the left, only prism C is moved one meter to the left. The present disclosure shares some features with traditional scene graphs, but may have additional functionality / behaviors as well: 1) the ability for indirect descendants of a node to be grouped with their indirect descendants (so that the node and indirect descendants undergo the same transformations if they are grouped), and / or 2) the ability to group two nodes that are on entirely separate transform trees, etc. In some embodiments, an indirect descendant may be a node that is not directly connected to a parent node, for example, possibly separated by two or more intervening nodes. In some embodiments, the grouping logic may be implemented by creating a root group node for all prisms that belong within the current group. If any prisms in the same group are descendants of other prisms in the group, they are also descendants of those prisms in the transform group tree. Note that in some embodiments, a separate data structure is not created for groups; instead, groups may have an additional flag attached to prisms.In some embodiments, if any prism in a group is transformed, a transformation may be computed and applied to the root group node. This transformation may propagate up the transform tree like any other transformation, with the transformation propagating not only to each child in the transform tree, but also to each prism in the group.

[0200] The following are various embodiments of adding and removing nodes from a group tree. Figures 10C-10E illustrate one embodiment for creating a new group. With reference to the transform tree 1020 in Figure 10C, this embodiment will illustrate steps for grouping together nodes A and E, which are currently in separate transform trees. (1) A determination is made as to whether A or E are already in a group. Currently, neither A nor E have a group parent, and therefore neither is in a group. Therefore, a group root is created. In Figure 10D, a group tree 1022 with a group root 1024 is created. (2) As depicted in Figure 10E, the group parents of A and E are assigned to the group root 1024. (3) Determine whether E is an ancestor (or parent) of A. A has no parent, and therefore E is not an ancestor of A. (4) Determine whether A is an ancestor (or parent) of E. E has no parent, and therefore A is not an ancestor of E.

[0201] 10F-10I illustrate another embodiment for creating a new group. With reference to the transform tree 1030 in FIG. 10F, this embodiment illustrates the steps of grouping nodes A and C together, with node C being a child node of node A as depicted in the transform tree 1030. (1) A determination is made as to whether A or C is already in a group. Currently, neither A nor C has a group parent, and therefore neither is in a group. Therefore, a group root is created. In FIG. 10G, a group tree 1032 is created with a group root 1034. (2) The group parent of A and C is the group root 1034. As depicted in FIG. 10H, A and C are assigned to the group root 1034. (3) Determine whether C is an ancestor (or parent) of A. A has no parent, and therefore C is not an ancestor of A. (4) Determine whether A is an ancestor (or parent) of E. A is an ancestor of C. Therefore, we reassign C's group parent to A.

[0202] 10J-10K illustrate an embodiment of adding an ungrouped node to an existing group. With reference to the transform tree 1040 and group tree 1042 in FIG. 10J, this embodiment illustrates steps for grouping nodes C and E from the transform tree 1040 into the same group. (1) Determine whether C or E are already in a group. As depicted in FIG. 10J, E is in a group with A. (2) Determine the group parent of C: Find the parent or ancestor of C that belongs to the same group as E. Here, A is in a group with E. A is an ancestor of C. Assign A as the group parent of C (see FIG. 10K). (3) Determine whether C has an ancestor or parent of any other node in the group. C has no children. Processing stops.

[0203] 10L-10N illustrate another embodiment of adding an ungrouped node to an existing group. With reference to the transform tree 1050 and group tree 1052 in FIG. 10L, this embodiment illustrates steps for grouping nodes B and E from the transform tree 1050 into the same group (e.g., group tree 1052). (1) Determine whether B or E is already in a group. Here, E is in a group with A and C. (2) Determine the group parent of B by finding the parent or ancestor of B that belongs to the same group as E. Here, A is in a group with E and is the parent of B. Assign B's group parent to A (see FIG. 10M). (3) Determine whether B is a parent or ancestor of any of the other nodes in the group. Here, C is a child or descendant of B. Change C's group parent to B (see FIG. 10N).

[0204] 10O-10R illustrate another embodiment of adding an ungrouped node to an existing group. With reference to a transform tree 1060 and a group tree 1062, this embodiment illustrates steps for grouping nodes A and E from the transform tree 1060 into the same group (e.g., group tree 1062). (1) Determine whether A or E are already in a group. Here, E is in a group with B and F. (2) Determine A's group parent by finding A's parent or ancestor that belongs to the same group as E. Here, A has no parent, so A is assigned to the group root 1064 (see FIG. 10P). (3) Determine whether A is a parent or ancestor of any of the other nodes in the group. Here, A finds B as a child. Therefore, B sets its group parent to A (see FIG. 10Q). In addition, A finds F as a descendant. A and F belong to the same group (e.g., group tree 1062). Therefore, F sets its group parent to A (see Figure 10R).

[0205] 10S-10W illustrate an embodiment of adding a grouped node to another group. With reference to the transform tree 1070 and group tree 1072 in FIG. 10S, this embodiment illustrates the steps of grouping nodes C and G into the same group. (1) Determine whether C or G are already in a group. Here, C is in a group with D. G is in a group with B and F. (2) Since both C and G are in a group, merge the two into a single group by arbitrarily choosing the final group to be the group of C and D (e.g., group root 1074). (3) Add only the direct children of the old group root 1076. This simply means adding B and F to C's group. (4) Add B to C's group. (4a) Determine B's group parent by finding B's parent or ancestor that belongs to the same group as C. Here, there is no group parent. Therefore, add B as a child of group root 1074 (see FIG. 10T). (4b) Determine whether B is a parent or ancestor of any of the other nodes in the group. Here, B finds C as a child. B and C belong to the same group. C sets its group parent to B (see Figure 10U). (5) Add F to C's group (Note: this is not arbitrary; G will not be selected to be added next, it will simply be pulled in along with F). (5a) Determine F's group parent by finding F's parent or ancestor that belongs to the same group as C. Here, F finds D as its parent. F sets its group parent to D (see Figure 10V). Note that G is automatically placed in the same group. (5b) Determine whether F is a parent or ancestor of any of the other nodes in the group. Here, F finds G, but G's parent is already F. (6) Delete the old group root 1076 (see Figure 10W). (Group conversion)

[0206] In some embodiments, any transform applied directly to a node that belongs in a group may transform the group along with it. Indirect transforms applied to a group node may not transform the group. For example, if a parent node is not in a group, but its child is in a group, and the parent is transformed, the nodes in the child's group will not transform along with it, even if the child's transform changes. If moving an entire group is desired, the parent should be added to the group.

[0207] The transform used to compute the transform to apply to the group must be applied to each immediate child of the group root.

[0208] 10X-10Y illustrate sample computations according to some embodiments of the present disclosure. For a given transform tree 1080 and group tree 1082 as depicted in FIG. 10X, the present embodiment applies a transform to node A. Because A belongs to the group tree 1082, nodes C, D, and E are expected to be transformed along with node A. In addition, node B, which does not belong to the group, must also be transformed along with the group because it is a child of A. Let A's new world matrix be denoted as Maw', where Maw' stands for "matrix a world new." Let A's old world matrix be denoted as Maw, where Maw stands for "matrix a world old." Let A's matrix to apply to A to obtain Maw' be denoted as Mgw, where Mgw stands for "matrix group world." The goal is to find a matrix that can be set at the group root (Mgw) that, when applied to node A's world matrix, will equal node A's new matrix. This same matrix is also applied to other immediate children of the group root.

[0209] Figure 10Y illustrates a sample calculation for applying a transformation to node A. From Figure 10Y, we focus on node A since that is the node whose matrix is to be changed, i.e.:

[0210] Maw'=Mgw×Maw

[0211] Maw' and Maw are known, solve Mgw

[0212] Mgw=Maw'×Maw^-1

[0213] Therefore, for each child node of the group root, set the node's world transformation to: Child Node New World Matrix = Mgw × (Child Node Old World Matrix). For this example, apply Mgw to the two direct children of the group root, A and E. Maw'=Mgw×Maw =(Maw'×Maw^-1)×Maw =Maw' Mew' = Mgw × Mew =(Maw'×Maw-1)×Mew

[0214] Note that the final world transformation of A is the desired Maw'. As the transformation propagates up the transformation tree, all child nodes (whether they are part of the same group or not) are updated. This means that node B is also updated.

[0215] 10Z-10AA illustrate sample computations according to some embodiments of the present disclosure. For a given transform tree 1090 and group tree 1092 as depicted in FIG. 10Z, the present embodiment applies a transform to node C. Because C belongs to the group, nodes A, D, and E are expected to be transformed along with it. In addition, node B, which does not belong to the group, must also be transformed along with the group because it is a child of A. Let us denote the new world matrix of C as Mcw', where Mcw' stands for "matrix c world new." Let us denote the old world matrix of C as Mcw, where Mcw stands for "matrix c world old." Let us denote the matrix applied to c to obtain Mcw' as Mgw, where Mgw stands for "matrix group." The goal is to find a matrix that can be set at the group root (Mgw) that, when applied to the world matrix of node C, will equal the new matrix of node C. This same matrix is also applied to other immediate children of the group room.

[0216] Referring to Figure 10AA, we focus on node C because that is the node whose matrix is being changed. Therefore, Mcw' = Mgw x Mcw. Since Mcw' and Mcw are known, we solve for Mgw. Mgw = Mcw' x Mcw^-1. So for each child node of the group root, set the world transformation of the node to be "child node new world matrix = Mgw x (child node old world matrix). For this example, remember what Mcw is equal to (Mcw = Ma1 x Mb1 x Mc1). Combining that with Mgw = Mcw x Mcw^-1, we get Mgw = Mcw' x (Ma1 x Mb1 x Mc1)^-1, which is Mgw = Mcw' x Mc1^-1 x Mb1^-1 x Ma1^-1. Remember that A and E are root nodes, so Maw = Ma1 and Mew = Me1. From there, we can find the new transformations for all nodes. Apply Mgw to the two direct children of the group root, E and A. (E is the transformation tree root node, so Mew==Me1). Mew' = Mgw × Mew =(Mcw'×Mc1^-1×Mb1^-1×Ma1^-1)×Mew =(Mcw'×Mc1^-1×Mb^-1×Ma1^-1)×Mel (A is the transformation tree root node, therefore Maw==Ma1) Maw'=Mgw×Maw =(Mcw'×Mc1^-1×Mb1^-1×Ma1^-1)×Maw =(Mcw'×Mc1^-1×Mb1^-1×Ma1^-1)×Ma1 (A is the parent of other nodes, so it updates the world transformation of its children nodes B and C) Mbw'=Maw'×Mb1 =(Mcw'×Mc1^-1×Mb1^-1)×Mb1 =Mcw'×Mc1^-1 Mcw'=Mbw'×Mc1 =(Mcw'×Mc1^-1)×Mc1 =Mcw' Note that the final world transformation of node C is the desired Mcw'.

[0217] In some embodiments, the transform tree and transform group functionality may be implemented within the universe itself. In some embodiments, other applications (e.g., application 140, icon grid application 260, status bar app 270, social panel app 280, store panel app 290, etc.) may not need this type of functionality, as the universe may manage this functionality. Other suitable methods or systems utilizing group trees and / or transform groups may be used. (Installed Launcher)

[0218] In some embodiments, a user may be able to place launchers in their landscape from which they want to launch applications or pin the launcher persistently. This allows the launcher to be a natural extension of the user's environment, creating a clear connection between the digital and physical worlds. A pinned launcher in its open state may have the same layout and structure as a regular launcher, except that it is attached to a location and may be expanded and / or contracted. In some embodiments, when a user creates a pinned launcher, the user has the option to pin the launcher persistently to that location. In this case, the launcher may remain in that location until the user moves or removes it. A regular launcher may still be launched when the user presses the home button, even if the launcher is pinned. In some embodiments, a user may pin multiple launchers (so, for example, the user may place them in a location where they most want to launch content). A pinned launcher may show recent applications launched from that location. This allows each launcher to be tailored to the space in which it resides. In some embodiments, the behavior of pinning a launcher application to a location is simulated by an Extractable Content Manager (ECM), which gives the appearance of a pinned launcher icon by literally copying the icon into a separate prism and allowing the user to move that prism.

[0219] In some embodiments, the launcher may be an application, and the launcher carousel and launcher menu panels may be presented inside separate prisms, so that the prisms themselves may be placed within the user's landscape.

[0220] 11 shows an exemplary view of a pinned launcher according to some embodiments. As shown in FIG. 11, a launcher menu 1100 may include an icon grid panel 1120, a minimized social panel 1110, and a minimized store panel 1130. The icon grid panel 1120 may actively display icons of applications that a user can launch, with the minimized social panel 1110 to the left of the icon grid panel 1120 and the minimized store panel 1130 to the right. An image icon 1140 may indicate to the user that the launcher has been pinned to a particular location (in this example, on a chair). A different image icon 1140 may indicate a pinned launcher as opposed to a pinned launcher. In some embodiments, a different number of panels and / or different content for the panels may be used.

[0221] In some embodiments, the pinned launcher menu may be persistently stored and remain pinned in that location until the user moves or removes the launcher from that location. Even if the user shuts down the mixed reality system, upon restarting the mixed reality system, the launcher menu may appear in that location after the system has fully launched if a set of criteria is met (such as the user being in proximity to the pinned launcher menu). In some embodiments, an application launched from a pinned or grounded launcher may be generated and displayed in the same location as the pinned or grounded launcher. In some embodiments, if the user selects a landscape application, the launcher may disappear and the application (e.g., Prism) may appear in the same location.

[0222] If the user chooses to pin the launcher to that location permanently, the launcher may remain there until the user moves or removes the launcher from that location, and the user can interact with the launcher and launch applications from that location. If the user does not interact with the launcher, it may enter its idle minimized state. When the user is not interacting with the launcher, the state of the pinned or pinned launcher may be changed to a minimized / idle state. From here, the pinned launcher may show the time, featured store content, featured social contacts, branded opportunities / experiences, and / or other content, etc.

[0223] FIG. 12 shows exemplary types of content that may be displayed while a stationary launcher is in an idle / sleep state according to some embodiments. The standard view of the stationary launcher in its minimized / sleep state may display a time 1210 showing the current time and date. When a user hovers over the stationary launcher using their head pose and / or clicks / selects the stationary launcher, the launcher changes state from sleep to active and renders, bringing up its full active view, e.g., showing at least the icon grid and corresponding panels. In some embodiments, main store content 1220 may be displayed in place of the standard view time and date 1210. Main store content 1220 may be pre-downloaded previews of applications that give a quick interactive glimpse of products. Other types of content that may be displayed may include, for example, engaging experiences such as children's toys 1230 or balloons with daily messages 1240. These engaging experiences may include content and creativity for device users. Artists, musicians, and information publishers may create this content. (Secondary UI - application options displayed within a carousel)

[0224] FIG. 13 illustrates an exemplary secondary UI volume according to some embodiments. Application options menu 1310 may indicate different interactions a user may have with content provided by an application rendered in a prism. Application options menu 1310 may be displayed within the three-dimensional space of secondary UI volume 220 from FIG. 2. In some embodiments, the application options menu may be a carousel with a central object always highlighted. As a user swipes to the next item, icons may become more visible and display appropriate text describing the options. Application title 1320 may help a user recognize the application providing the content. Here, as an example, the application is a 2D Internet browser displaying a web page. Settings 1330 may display application settings and / or contextually link to a universe settings application. Close application 1340 may close / remove an application from the user's landscape. Share 1350 may allow a user to initiate a sharing session with other users of this application. Download 1360 may download content displayed in the prism. For example, a user may wish to download an article that is displayed within a 2D internet browser. Those skilled in the art will appreciate that the application options menu 1310 may include more or fewer options and may be in a different order than that disclosed and shown. (body dynamics)

[0225] Body dynamics refers to the way in which displayed virtual content anchored to the user's body can be managed and presented to the user.

[0226] 14 shows examples of body movements according to some embodiments. Body movements may refer to the way a user application / user interface / content 1480 moves in relation to a user 1490. There are different body movement types, which may have different types of user interface effects. Body movement types may also be referred to as anchoring types, which refer to the way a prism and / or an application within a prism is anchored and displayed within the user's landscape.

[0227] FIG. 15 illustrates different types of body movements according to some embodiments. One example of a type of body movement is world-fixed 1520, which means there are no body movements. In some embodiments, a generic prism may be world-fixed and have no body movements unless the user selects to activate follow behavior from the application options. Another example of a type of body movement is billboarding 1530. In some embodiments, billboarding maintains a fixed orientation and / or angle relative to the user as the angle and / or orientation of the application content changes. The relative angle and / or orientation may be determined in any number of ways. For example, virtual content may have a predefined forward or frontal direction. The system may require the forward vector of the virtual content to point toward the user as determined by head pose. In some embodiments, billboarding behavior maintains a fixed location in 3D space, while other embodiments may allow the location to vary. In some embodiments, billboarding behavior may also include edge billboarding 1540, which begins turning toward the user only if the user would be viewing the content from an excessively oblique angle. For example, the application remains stationary until the user is viewing a web page from a side angle that makes it difficult to read the text. The application projects a frustum from the front planar surface of the prism that follows the edge of where the user presses. Parameters (e.g., application-specific parameters) for edge billboarding may include a frustum angle, a following behavior such as centering the user or moving the frustum to the edge of where the user presses, a force field (push) Boolean value, a force field activation distance, speed, and / or an inverted Boolean value.

[0228] The Follow 1550 body motion type is another type of body motion. In some embodiments, Follow 1550 is a specific mode in which a single object (e.g., a prism) can be placed by the user, e.g., temporarily attaching the object to the user's head position. This is useful when the user wants the object to move with their head position. A feature of Follow is that it ignores the pitch (e.g., up / down rotation) of the user's head, giving the user more degrees of freedom to look around their physical space. If the user looks left or right, the object may re-center within its field of view. In some embodiments, this re-centering has a linear interpolation value, which takes time to move and feels somewhat loosely attached to the user. In some embodiments, the Follow behavior may directly mimic the user's head movement, e.g., one-to-one movement. In some embodiments, Follow can be accessed and activated for each prism through an application options menu. In some embodiments, the tracking mode may be a default assignment to the prism and / or may be a non-temporary attribute of the prism (e.g., fixed as a tracking prism until the user, for example, appropriately changes the prism).

[0229] In some embodiments, the body motion type of Follow 1550 can have one of two settings, such as Passive Head Fixation 1560 and External Sensor 1570. During the Passive Head Fixation 1560 setting, a single object, prism, or application can be attached to the user's head position, similar to the Follow 1550 setting; however, the pitch parameter may not be fixed by the user, so that the object can follow similarly if the user looks up or down. During the External Sensor 1570 setting, the object, prism, or application can follow the user based on data from an external sensor. In some embodiments, an inertial measurement unit (IMU) can be included in the user's beltpack unit. In some embodiments, the IMU is a unit within the electronics module that collects angular velocity and linear acceleration data, which is sent to the main processor. The IMU housing can include two separate sensors. The first sensor can be an accelerometer with three-fold symmetry. This can generate three analog signals describing the acceleration along each of its axes produced by and acting on the user. The second sensor could be a three-fold angular velocity sensor. It could also output three analog signals. These signals could describe the user's angular velocity about each of the sensor axes. This IMU would allow a mixed reality system to have a prism relative to the body instead of needing to be relative to the user's head pose.

[0230] For a Fading 1580 body dynamics setting, at a specified distance before the clipping plane, content may begin to dissipate / disappear. At a specified distance before the clipping plane, in some embodiments, the mixed reality system may no longer comfortably render the content, and / or as the user approaches that cutoff point, the content should smoothly fade out rather than abruptly turning off. In some embodiments, fading may occur as the user approaches the clipping plane (e.g., the exact measurement may be hardware-dependent). Virtual content in the landscape may have this effect as the user passes through or is very close to the content. (Simultaneous launch and installation of applications)

[0231] In some embodiments, it may be beneficial to simultaneously launch and place applications in a mixed reality environment. In one embodiment, each time a cursor moves over a web page, the browser engine may perform a hit test on the web page content to see what is under the cursor position. The hit test may detect elements within the web page and then analyze the type of element. Every html element may have different parameters, some of which are common to all elements, some of which are common to multiple elements, and some of which are element-specific. For links, the most common and most recommended element is the stickier tag.<ahref=’mylink’> The use of the hit test tag is described below. This hit test tag can be used with any element, image, or text. At any point in time, the type of element the cursor is currently over can be requested. The browser engine can cache the results of the hit test and pass the node or node type to the browser application. In some embodiments, this caching of the hit test results can be performed when a user selects a link using an extract trigger.

[0232] In another embodiment, when the cursor is moved over any element in a web page that is clickable, the web engine may call an API to the browser application requesting that the cursor change from an "arrow" or "i-beam" to a "hand." When this API is called, the browser application may then request information about the type of element the cursor is over.

[0233] FIG. 16 shows a flowchart for simultaneously launching and placing applications in a mixed reality environment according to some embodiments. At 1610, a user may move a cursor over a link from a web page displayed in a browser application displayed inside a prism in the user's landscape. The cursor may be moved either by a mouse, a handheld controller device such as a totem, or by eye gaze, in which the user's eyes are centered on the web page and sensors from the user's head-mounted system may detect that the user is viewing a link from the web page, or any other suitable method. The link may be a uniform resource locator (URL). Moving the cursor over the link and / or an eye gaze centered on the web page having the link (or any other suitable user selection method) may send user input to the application rendering the content indicating that the user may be interested in the content corresponding to the link.

[0234] At 1615, the web engine and / or browser application may detect a link that the user navigates to using the cursor or eye gaze. Upon detecting the link, the web engine may notify the browser application that the user may be interested in content, which may cause the browser application to request Universe to create a mini display volume (e.g., a mini prism). The mini prism may be the initial default size of a standard 3D display volume management unit (e.g., a prism) for the purpose of serving as a placeholder prism for the browser application when it loads content from the link into the mini prism.

[0235] At 1620, Universe creates a mini-prism, loads it, and begins showing a page preview of the loaded content. At 1625, the user may use an extract trigger to select a link and indicate that they are interested in viewing the content to which the link points. Use of an extract trigger may indicate to the application that the user may be highly interested in viewing the content corresponding to the link. The browser may cache the results of the hit test and pass the node or node type to the browser application for loading the prism. At 1630, the user may use a handheld controller to move the mini-prism from the web page. This movement of the mini-prism may send user input to Universe indicating that the user is moving the mini-prism.

[0236] At 1635, the mini prism is in place mode, and the user may move the mini prism and place it at a location in the user's landscape. At 1640, the user may release the mouse button or extraction trigger and place the mini prism at a location in the user's landscape. Releasing the mouse button or extraction trigger may send user input to the universe indicating that the mini prism has been placed at a location in the augmented reality environment (e.g., the user's landscape).

[0237] At 1645, the mini-prism may be expanded to its normal size (e.g., not the mini-prism size) in place, with the page of links fully loaded within the prism for display to the user. In effect, the user can simultaneously open / launch and place an application within the augmented reality environment by selecting a link and placing the link's mini-prism within the mixed reality environment, while in the background the mixed reality system loads and prepares the prism for display.

[0238] In an alternative embodiment, at 1670, after 1615, the user may select a link using an extraction trigger before the mini-prism is requested by the application and created by the universe at 1675. This may help reduce system processing to create the mini-prism before the user actually decides to extract content from the link for display.

[0239] In an alternative embodiment, after the user selects a link with the extract trigger, the user may release the extract trigger without moving the mini-prism at 1680. In this embodiment, instead of expanding the mini-prism to its standard size with the prism fully loaded with the content from the link at 1685, the universe may simply create a new tab in the web browser application using the link, with the content of the link already loaded and cached when the user selects the link with user input equivalent to the extract trigger.

[0240] The disclosed subject matter is a system and method for managing and displaying virtual content within a mixed reality environment. In particular, the system and method disclose a Universe application that can act as a 3D window manager by managing 3D windows such as prisms. A prism is a bounded volume positioned in space (e.g., by a user). Applications can render graphics into the prisms through the Universe. The Universe works with Kali to render the scene graph and has full control over the placement, scaling, etc. of each prism. In some embodiments, the Universe can provide the ability to attach prisms to walls and surfaces, register prisms with a passable world system, and / or control the sharing of content among multiple users of the mixed reality system. In some embodiments, the universe also manages the prisms themselves (e.g., providing basic user interface controls for creating prisms, managing placement and / or snapping rules, i.e., close button, action bar, etc.), disregarding the content inside the prism, and / or keeping track of the prism (e.g., the application that owns the prism, the location to place the prism, how the prism is anchored, i.e., body-centered, world-fixed, etc.). In some embodiments, prism behavior may be based in part on placement. In some embodiments, prism behavior may be based in part on placement (e.g., user placement through user interaction) and / or body dynamics (e.g., billboarding, body-centered, passive head-fixed, etc.).

[0241] As a 3D window manager, Universe may provide more functionality than a standard 2D window manager. In some embodiments, one or more additional features (e.g., functionality) that Universe may provide as a 3D window manager (e.g., beyond a 2D window manager) may include:

[0242] Persistence: An application located on a user's kitchen counter may appear on the kitchen counter unless the user changes it. The user may not need to relaunch the application every time the system is turned on / off or every time the user leaves the room and returns. Because the universe stores prism information in the passable world system, the universe may resume an application located on the user's kitchen every time the user uses the mixed reality system and comes into close proximity to the application in the user's kitchen.

[0243] Application state to the user: Applications start, suspend / pause, and resume automatically, i.e., no explicit user action is required. This contrasts with 2D window managers, where user interaction is required to change the application's operational state (e.g., the user clicking a close button).

[0244] Locality - Real World vs. 2D Window: 2D window managers may be constrained to a screen regardless of where the screen may be placed. However, within a mixed reality environment, applications may be placed relative to the real world, for example, relative to other things such as physical objects in the user's environment, the user's body, fixed locations, etc.

[0245] Physics: Prisms can move, so movement and tracking of the prism needs to be managed (e.g., billboarding / body centering relative to the user, passive billboarding, bouncing when moving, bump bounces, etc.).

[0246] Private but Interactive: 2D windows are private but they are not interactive, or they are interactive but not private. However, a universe may enable both privacy and interactivity for 3D windows (e.g., prisms). Additional Embodiments

[0247] Additional embodiments of the present disclosure are described below. These additional embodiments may incorporate elements from the embodiments disclosed above. 1. A method for initiating a mixed reality system, the method comprising: Determining a user's current location; Retrieving one or more prisms previously deployed at the current location; Restoring one or more prisms at the user's current location; Displaying the restored one or more prisms at the user's current location; and A method comprising: 2. The method of embodiment 1, wherein the prism is a volume of space that contains virtual content from the application. 3. The method of embodiment 2, wherein the application is an application instance of an application when the application renders in two or more Prisms. 4. The method of embodiment 1, wherein the prism represents a portion of a scene graph, the scene graph corresponding to a user's current location. 5. The method of embodiment 4, wherein the scene graph comprises data from the first application and the second application. 6. Reading one or more prisms that were previously deployed at the current location Retrieving instance data for one or more prisms from an external database; Reconstructing the local prism database with instance data for one or more prisms; 2. The method of embodiment 1, comprising: 7. The method of embodiment 6, wherein the instance data for each prism includes a prism attribute data structure defining the prism, the prism attributes comprising at least one of location, orientation, range width, range height, range depth, body motion, anchoring type, or anchoring position. 8. The method of embodiment 7, wherein the instance data for each prism includes application-specific attribute data comprising state information for virtual content previously rendered in the prism by the application. 9. Restoring one or more prisms is launching each application corresponding to one or more Prisms previously deployed at the current location; and creating one or more new prisms corresponding to one or more previously deployed prisms; 2. The method of claim 1, further comprising: rendering each virtual content into one or more new prisms. 10. The method of embodiment 9, further comprising displaying placeholder prisms in locations corresponding to the locations of one or more previously deployed prisms, and wherein displaying the one or more prisms further comprises replacing each placeholder prism with one or more new prisms having respective virtual content. 11. When a user interacts with one or more Prisms, updating the user's local Prism database with updated Prism instance data; Synchronize your local Prism database with an external database 2. The method of embodiment 1, further comprising: 12. A method for displaying virtual content in a 3D spatial environment, the method comprising: receiving a request from an application to display virtual content in a 3D spatial environment; creating a prism, the prism being a volume of space configured to confine the boundary of the virtual content within the boundary of the prism; receiving virtual content from an application; Rendering the virtual content within the boundary of the prism; and Associating prisms with objects in a 3D spatial environment A method comprising: 13. The method of embodiment 12, wherein the boundaries of the prisms are not displayed. 14. The method of embodiment 12, wherein the 3D spatial environment is the user's physical real-world environment. 15. The method of embodiment 12, wherein the prism is automatically created and has a set of functionality. 16. The method of embodiment 15, wherein the set of functionality comprises an association between a prism and an object in a 3D spatial environment. 17. The method of embodiment 15, wherein the set of functionality comprises one or more of a minimum size allowed for the prism, a maximum size allowed for the prism, and an aspect ratio for sizing the prism. 18. The method of embodiment 12, wherein the application renders first virtual content in a first prism and second virtual content in a second prism, the first prism and the second prism being different prisms. 19. The method of embodiment 18, wherein the prisms do not overlap with other prisms in the 3D spatial environment. 20. The method of embodiment 12, wherein the prism is placed relative to the object in a 3D spatial environment. 21. The method of embodiment 20, wherein the object is a user of the augmented reality device. 22. Prisms are One or more universal characteristics; one or more application-specific features; 13. The method of embodiment 12, wherein the one or more universal features and the one or more application-specific features are selected from a list of pre-approved options. 23. The method of embodiment 22, wherein the one or more universal features ensure that different applications interact with each other in a consistent manner. 24. A method for managing application state of virtual content in a mixed reality system, the method comprising: Dividing a 3D volume into a volumetric grid, determining a first location of a user within a volume grid; Determining a second location of the application within the 3D volume; Calculating a distance between a user and an application within a 3D volume; modifying a state of the application based at least in part on the calculated distance between the user and the application; A method comprising: 25. The method of embodiment 24, wherein the radius of the active zone defines a circular / spherical area around the user using the mixed reality device. 26. The method of embodiment 25, further comprising a buffer zone outside and surrounding the active zone, the buffer zone preventing intermittent or rapid changes in the state of the application. 27. The method of embodiment 24, wherein modifying the state of the application is based at least in part on whether the application is blocked by another application. 28. The method of embodiment 24, wherein modifying the state of the application is based at least in part on the user's head pose. 29. The method of embodiment 24, wherein the distance of the known location of the application within the cell is determined only with respect to the cell in which the user using the mixed reality device is present and its neighboring cells. 30. A method of opening and placing an application in an augmented reality environment, the method comprising: receiving a first user input from a user indicating a request for new content; launching an application and generating the content; creating a mini-display volume of the 3D display volume management unit, wherein the page preview is displayed in the mini-display volume, and the 3D display volume management unit is created upon launch of the application; receiving a second user input indicating movement of the mini-display volume; receiving a third user input indicating placement of the mini-display volume at a location in the augmented reality environment; Expanding the 3D display volume management unit in place of the mini display volume; Including, The 3D display volume management unit displays content that is fully loaded within the 3D display volume management unit. 31. The method of embodiment 30, wherein the first user input is cursor movement over a link on a web page. 32. The method of embodiment 31, wherein the second user input is the selection of a link and the movement of the mini-display volume. 33. The method of embodiment 30, wherein the 3D display volume management unit replaces the mini-display when the 3D display volume management unit expands in place of the mini-display. 34. The method of embodiment 30, wherein content is loaded into the 3D display volume management unit while the user is moving the mini-display. 35. The method of embodiment 30, wherein the location is fixed to an object within the augmented reality environment. 36. The method of embodiment 35, wherein the object is a user. 37. A method for managing virtual content, the method comprising: receiving content from a content generation application; displaying content in a 3D spatial environment via a universe application; The universe application continuously manages the display of content in a 3D spatial environment. A method comprising: 38. A system, method, and computer program product for managing and displaying virtual content in a mixed reality system in accordance with any of the inventive concepts disclosed herein. 39. The system, method, and computer program product of embodiment 38, further comprising displaying the virtual content within a portion of the available 3D displayable space by displaying the virtual content within the stereoscopic display space, wherein the boundaries of the stereoscopic display space are not displayed. 40. The system, method, and computer program product of embodiment 38, further comprising assigning universal features and application-selected features to the prism from a list of pre-approved options for configuring display customization by the application. 41. The system, method, and computer program product of embodiment 38, further comprising displaying the virtual content within one or more prisms, wherein the one or more prisms do not overlap with one another. 42. The system, method, and computer program product of embodiment 38, further comprising varying the state of the prism based at least in part on the relative position and location of the prism with respect to the user. 43. The system, method, and computer program product of embodiment 38, further comprising managing content creation in the application and managing content display in a separate application. 44. The system, method, and computer program product of embodiment 38, further comprising opening an application that will provide content into the prism while placing the prism in the mixed reality environment. 45. The system, method, and computer program product of embodiment 38, further comprising assigning location, orientation, and range data to the prism to display virtual content within the prism, wherein the virtual content is 3D virtual content. 46. A system, method, and computer program product as described in embodiment 45, wherein the location is the coordinates of the prism's fixed position in the mixed reality environment. 47. The system, method, and computer program product of embodiment 45, wherein the orientation defines how the prism is rotated relative to the object, the object being a wall. 48. The system, method, and computer program product of embodiment 45, wherein the range data defines the size of the prism. 49. The system, method, and computer program product of embodiment 38, further comprising pinning the launcher application to a real-world object in the mixed reality environment. 50. The system, method, and computer program product of embodiment 49, wherein the pinned launcher application launches the content of the application in Prism in the same location as the pinned launcher application. 51. The system, method, and computer program product of embodiment 38, further comprising assigning a behavior type to each prism, the behavior type comprising at least one of world-fixed, billboarding, edge-billboarding, following head-fixed, following based on external sensors, or fading. 52. The system, method, and computer program product of embodiment 38, further comprising identifying the most used content specific to the installed location of the launcher application. 53. The system, method, and computer program product of embodiment 38, further comprising displaying favorite applications by the installed launcher application, the favorite applications being based at least in part on context relative to the location of the installed launcher. 54. A method, comprising: accessing a scene graph for a scene, the scene graph comprising one or more transformation trees, each tree comprising a plurality of nodes; adding tags to one or more nodes from a plurality of nodes in one or more transform trees, the one or more tagged nodes forming a transform group, the one or more tagged nodes comprising a first tagged node and a second tagged node; moving the first tagged node; Including, A method wherein moving a first tagged node causes a second tagged node to move. 55. The method of embodiment 54, wherein the second tagged node is not a direct descendant of the first tagged node. 56. The method of embodiment 54, wherein the first tagged node and the second tagged node are not from the same transformation tree. 57. The method of embodiment 54, wherein the plurality of nodes comprises a first node and a second node, and the method further includes parenting the first node to the second node, and parenting the first node to the second node does not move the parented node. 58. A method for displaying virtual content in a 3D shared space, the method comprising: generating, by a first application, virtual content in a 3D shared space; displaying, by the second application, the virtual content generated by the first application; Including, The method, wherein the first application and the second application are different applications. (System architecture overview)

[0248] 17 is a block diagram of an exemplary computing system 1400 suitable for implementing one or more embodiments of the present disclosure. The computing system 1400 includes a bus 1406 or other communication mechanism for communicating information, which interconnects subsystems and devices such as a processor 1407, a main memory 1408 (e.g., RAM), a static storage device 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., a modem or Ethernet card), a display 1411 (e.g., CRT or LCD), an input device 1412 (e.g., a keyboard), and cursor control.

[0249] According to some embodiments, computing system 1400 performs certain operations by processor 1407 executing one or more sequences of one or more instructions contained in main memory 1408. Such instructions may be read into main memory 1408 from another computer-readable / usable medium, such as static storage device 1409 or disk drive 1410. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall refer to any combination of software or hardware used to implement all or part of the present disclosure.

[0250] The terms "computer-readable medium" or "computer-usable medium," as used herein, refer to any medium that participates in providing instructions to the processor 1407 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1410. Volatile media include dynamic memory, such as the main memory 1408. In some embodiments, the system may use solid-state drive (SSD) memory.

[0251] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0252] In one embodiment, execution of a sequence of instructions to practice the present disclosure is performed by a single computing system 1400. According to other embodiments, two or more computing systems 1400 coupled by a communications link 1415 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequence of instructions required to practice the present disclosure.

[0253] Computing system 1400 may transmit and receive messages, data, and instructions, including programs, e.g., application code, via communications interface 1414 through communications link 1415. Received program code may be executed by processor 1407 as it is received and / or stored on disk drive 1410 or other non-volatile storage for later execution. Computing system 1400 may communicate with a database 1432 on external storage device 1431 through data interface 1433.

[0254] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the process flows described above are described with reference to a particular order of process actions. However, the order of many of the described process actions can be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, therefore, to be regarded in an illustrative sense, and not in a restrictive sense.

Claims

1. 1. A method for managing application state of virtual content in a mixed reality system, the method comprising: Dividing the 3D volume into a volumetric grid; determining a first location of a user within the volume grid; determining a second location of the application within the 3D volume; calculating a distance between the user and the application within the 3D volume; modifying a state of the application based at least in part on the calculated distance between the user and the application; and Including, A method, wherein modifying the state of the application is based at least in part on whether the application is obscured by another application.

2. The method of claim 1 , wherein the radius of the active zone defines a circular / spherical area around the user using the mixed reality device.

3. The method of claim 2 , further comprising a buffer zone outside and surrounding the active zone, the buffer zone preventing intermittent or rapid changes in the state of the application.

4. The method of claim 1 , wherein modifying the state of the application is based at least in part on a head pose of the user.

5. The method described in claim 1, wherein the user is located within a cell of the three-dimensional grid and the distance of a known location of an application within the cell is determined only with respect to the cell and with respect to neighboring cells of the cell.

Images